Bolog
0%

COSBench搭建及使用

Posted on Edited on In

cosbench-usage.png

COSBench是Intel团队基于java开发，对云存储的测试工具，全称是Cloud object Storage Bench。COSBench也分控制台和发起请求的driver，且driver可以分布式部署。可以支持swift、s3、Openstack等接口

获取

从github上下载release版本（https://github.com/intel-cloud/cosbench）。解压后可以获取COSBenchUserGuide.pdf（只有英文版）和workload配置文件（./conf/*.xml）。

部署&启动

COSBench的运行依赖于java和curl这两个包，并且调用linux的nc来做数据分析，如果没有安装nc，请手动安装。
本人将COSBench部署到docker中，为的是方便以后运行，不要将有限的生命耗费到无限的折腾当中。

docker镜像使用的是centos:7.3.1611，具体dockerfile如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM centos:7.3.1611

RUN yum update -y
RUN yum install -y java-1.8.0-openjdk

RUN yum clean all

EXPOSE 19088 18088

ADD ./cosbench.tar.gz /root/
ADD ./entry.sh /root/

ENTRYPOINT ["/root/entry.sh"]

entry.sh是容器的启动脚本，脚本中主要用于启动COSBench服务，具体实现如下：

1
2
3
4
5
6
7
#!/bin/sh

./start-all.sh

while true; do
    sleep 10s
done

完成docker镜像的生成后，使用docker run -d -w /root/cosbench/ --rm -P s3test:performance命令运行容器。然后通过docker ps查看可访问的端口。

1
2
3
$ docker ps
CONTAINER ID        IMAGE                COMMAND             CREATED             STATUS              PORTS                                                NAMES
ed37db78178a        s3test:performance   "/root/entry.sh"    19 hours ago        Up 19 hours         0.0.0.0:32771->18088/tcp, 0.0.0.0:32770->19088/tcp   unruffled_mahavira

在浏览器中输入http://x.x.x.x:32770/controller/index.html访问COSBench首页。COSBench是以”master-slave”模式运行的，19088端口对应的是master端口，18088对应的是driver端口，任务的运行是有master下发给driver，然后由driver执行的（driver可配置多个）。

关于workload配置文件的撰写请见COSBenchUserGuide.pdf，此处给出一个测试S3接口的例子

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
<?xml version="1.0" encoding="UTF-8" ?>
<workload name="S3Test" description="Performance">
    <storage type="s3" config="accesskey=Ltiakby8pAAbHMjpUr3L;secretkey=qMTe5ibLW49iFDEHNKqspdnJ8pwaawA9GYrBXUYc;endpoint=http://los-cn-north-2.lecloudapis.com" /> 
    <workflow>
    
        <workstage name="init">
            <work type="init" workers="1" config="cprefix=testp;containers=r(1,4)" /> 
        </workstage>
        
        <workstage name="prepare">
            <work type="prepare" workers="1" config="cprefix=testp;containers=r(1,2);objects=r(1,800);sizes=c(4)MB" />
        </workstage>
        
        <workstage name="sequential_RO_8W_4MB">
            <work name="sequential_RO_8W_4MB" workers="8" totalOps="1600">
                <operation type="read" ratio="100" config="cprefix=testp;containers=c(1);objects=s(1,800);size=c(4)MB" /> 
            </work>
        </workstage>
        
        <workstage name="random_RO_8W_4MB">
            <work name="random_RO_8W_4MB" workers="8" totalOps="1600">
                <operation type="read" ratio="100" config="cprefix=testp;containers=c(2);objects=u(1,800);size=c(4)MB" /> 
            </work>
        </workstage>
        
        <workstage name="sequential_WO_8W_4MB">
            <work name="sequential_WO_8W_4MB" workers="8" totalOps="1600">
                <operation type="write" ratio="100" config="cprefix=testp;containers=c(3);objects=s(1,800);sizes=c(4)MB" /> 
            </work>
        </workstage>
        
        <workstage name="random_WO_8W_4MB">
            <work name="random_WO_8W_4MB" workers="8" totalOps="1600">
                <operation type="write" ratio="100" config="cprefix=testp;containers=c(4);objects=u(1,800);sizes=c(4)MB" /> 
            </work>
        </workstage>
        
        <workstage name="cleanup">
            <work type="cleanup" workers="1" config="cprefix=testp;containers=r(1,4);objects=r(1,800)" />
        </workstage>
        
        <workstage name="dispose">
            <work type="dispose" workers="1" config="cprefix=testp;containers=r(1,4)" />
        </workstage>
        
        <workstage name="init">
            <work type="init" workers="1" config="cprefix=testp;containers=r(1,4)" /> 
        </workstage>
        
        <workstage name="prepare">
            <work type="prepare" workers="1" config="cprefix=testp;containers=r(1,2);objects=r(1,800);sizes=c(64)KB" />
        </workstage>
        
        <workstage name="sequential_RO_32W_64KB">
            <work name="sequential_RO_32W_64KB" workers="32" totalOps="1600">
                <operation type="read" ratio="100" config="cprefix=testp;containers=c(1);objects=s(1,800);size=c(64)KB" /> 
            </work>
        </workstage>
        
        <workstage name="random_RO_32W_64KB">
            <work name="random_RO_32W_64KB" workers="32" totalOps="1600">
                <operation type="read" ratio="100" config="cprefix=testp;containers=c(2);objects=u(1,800);size=c(64)KB" /> 
            </work>
        </workstage>
        
        <workstage name="sequential_WO_32w_64KB">
            <work name="sequential_WO_32w_64KB" workers="32" totalOps="1600">
                <operation type="write" ratio="100" config="cprefix=testp;containers=c(3);objects=s(1,800);sizes=c(64)KB" /> 
            </work>
        </workstage>
        
        <workstage name="random_WO_32W_64KB">
            <work name="random_WO_32W_64KB" workers="32" totalOps="1600">
                <operation type="write" ratio="100" config="cprefix=testp;containers=c(4);objects=u(1,800);sizes=c(64)KB" /> 
            </work>
        </workstage>
        
        <workstage name="cleanup">
            <work type="cleanup" workers="1" config="cprefix=testp;containers=r(1,4);objects=r(1,800)" />
        </workstage>
        
        <workstage name="dispose">
            <work type="dispose" workers="1" config="cprefix=testp;containers=r(1,4)" />
        </workstage>
    
    </workflow>
</workload>

问题及解决

  1. COSBench首页时间显示没有按东八区显示
    在cosbench-start.sh中找到/usr/bin/nohup java这一行，增加-Duser.timezone=Asia/Shanghai参数
  2. COSBench在测试过程中MD5校验失败问题
    在测试过程中GET操作失败，而且错误日志如下： 
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    2017-07-11 07:26:42,297 [INFO] [Log4jLogManager] - will append log to file /root/cosbench/log/mission/MA308B012AE.log
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp2/myobjects38
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects43
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects43
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects31
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects7
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects26
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp1/myobjects34
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp2/myobjects25
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp2/myobjects45
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp1/myobjects7
    2017-07-11 07:26:42,574 [INFO] [NoneStorage] - performing GET at /testp2/myobjects43
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp2/myobjects16
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp2/myobjects23
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp1/myobjects15
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp1/myobjects36
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp2/myobjects34
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp1/myobjects34
    2017-07-11 07:26:42,576 [INFO] [NoneStorage] - performing GET at /testp2/myobjects27
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects49
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects27
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects24
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects35
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects2
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects47
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects39
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects50
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects8
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects21
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects24
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects15
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp1/myobjects4
    2017-07-11 07:26:42,575 [INFO] [NoneStorage] - performing GET at /testp2/myobjects1
    2017-07-11 07:26:45,476 [ERROR] [AbstractOperator] - worker 8 fail to perform operation testp1/myobjects7
    com.amazonaws.AmazonClientException: Unable to verify integrity of data download.  Client calculated content hash didn't match hash calculated by Amazon S3.  The data may be corrupt.
       at com.amazonaws.services.s3.internal.DigestValidationInputStream.validateMD5Digest(DigestValidationInputStream.java:79)
       at com.amazonaws.services.s3.internal.DigestValidationInputStream.read(DigestValidationInputStream.java:61)
       at com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:72)
       at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:155)
       at com.amazonaws.services.s3.model.S3ObjectInputStream.read(S3ObjectInputStream.java:147)
       at com.intel.cosbench.driver.operator.Reader.copyLarge(Reader.java:120)
       at com.intel.cosbench.driver.operator.Reader.doRead(Reader.java:92)
       at com.intel.cosbench.driver.operator.Reader.operate(Reader.java:69)
       at com.intel.cosbench.driver.operator.AbstractOperator.operate(AbstractOperator.java:76)
       at com.intel.cosbench.driver.agent.WorkAgent.performOperation(WorkAgent.java:197)
       at com.intel.cosbench.driver.agent.WorkAgent.doWork(WorkAgent.java:177)
       at com.intel.cosbench.driver.agent.WorkAgent.execute(WorkAgent.java:134)
       at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:44)
       at com.intel.cosbench.driver.agent.AbstractAgent.call(AbstractAgent.java:1)
       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:748)
    2017-07-11 07:26:45,476 [INFO] [NoneStorage] - performing GET at /testp1/myobjects14
    2017-07-11 07:26:46,880 [INFO] [NoneStorage] - performing GET at /testp2/myobjects14
    2017-07-11 07:26:49,806 [INFO] [NoneStorage] - performing GET at /testp1/myobjects43
    2017-07-11 07:26:51,514 [INFO] [NoneStorage] - performing GET at /testp2/myobjects31
    2017-07-11 07:26:54,021 [INFO] [NoneStorage] - performing GET at /testp1/myobjects5
    你可以通过增加-Dcom.amazonaws.services.s3.disableGetObjectMD5Validation=true参数来关闭MD5校验，据说此bug只在0.4.3.c4版本出现，我没验证过其它版本。

参考&鸣谢