What are /r/ceph's favorite Products & Services?

From 3.5 billion Reddit comments

ceph-deploy can be pointed at local repos or you can configure through a proxy. I would probably copy the repo to the deploy server if you aren't going to have access to any other boxes.

you should have an odd number of monitors (3). You can use the deploy box imo, it won't do anything most of the time.

There are also very good ansible playbooks to consider instead of ceph-deploy. https://github.com/ceph/ceph-ansible

I would house rados gateways on separate hosts. Same for mds. mon/mgr are really the only recommended ok for colocation.

you should have one osd per disk, not per server. so if you have 30 4tb disks, you would have 30 osds on that host.

2 networks a public and a private is a recommended setup, its not necessary though.

ceph using the default settings is basically raid1 across a failure domain you specify (host, rack, etc).

A lot of good information is at http://ceph.com/docs/master/start

I don't know how well MDS is at this point. So I can't say anything about 1)

If you want to do the VM-approach use libvirt's built-in support for RBD rather than using the kernel rbd and mapping inside. That's what I would have done. (So 2. essentially)

http://ceph.com/docs/master/rbd/libvirt/

Bummer, but thank you and thank you u/ctrl_alt_lynx

I actually had really bad luck with ebay Mellanox or Intel cards and knock-offs.

For my TrueNAS machine, I ended up with a X520-10G-1S 10Gek card and it's served me really well.

Ceph's great aggeregate performance comes from parallelization of disk's and hosts. and that you are adding cpu and ram (in effect cache) by every node added to the cluster.

if you skip all the cache tricks that are often used both in ceph, and big enterprise storage solutions. you always end up with data having to go to and from disks.

so when you look at replication pools. a single block/object of data have to be written to all 3 osd replicas. so the write speed is that of the slowest drive. But when doing deep io depth writes, the next object may (or may not, depending on osd count) go to another osd or host, so you spread your writing over more spindles and get higher bandwith. Or if you have many parallell writing threads you also get more spindles spinning and better aggeregate performance.

same basicaly happen when reading, except only the primary osd reads and delivers the data.

with erasure coding a piece of data is split, done magic math on and spread out over more disks. you loose performance and iops from overhead on erasure coding, but you get more spindles spinning on a given single sequencial read or write. so in some cases EC is "better" (for a limited subset of workload) then replicated pools.

so if you are testing a single 1 io depth 4k write. eg with dd or similar , you will never see much performance. The built in rados bench is a good tool to get ceph aggregate values.

there are some things to consider tho:
spinning bluestore disks should have ssd based DB disks; an enterprise SSD should have iops enough for 4-5 spinning disks. the default size of the SSD DB partitions may need tweaking depending on your osd and object size and count. Random access workloads would benefit a lot from beeing on flash based osd's tho.

when it comes to numbers you can gleen at other's that have done benchmarks on similar size clusters. https://www.slideshare.net/JoseDeLaRosa7/ceph-perfsizingguide is an example.

kind regards Ronny Aasen

I'm not sure if I see a real problem. Your tests are all with seemingly a large request size because troughput in MB/s is amazing and that would explain the low number of IOPs. Sequential and 'random' reads seems to bottleneck at the same MB/s so could that be networking? Check the interface of both the storage nodes and your client.

1500MB/s sounds like an odd number.

​

If you really want to test your cluster, use FIO for testing. https://github.com/axboe/fio

1. Create a RBD volume
2. Map the RBD volume on the RBD client (/dev/rbd0 maybe)
3. Use the raw mapped RBD device for FIO benchmarks, use randread/randwrite + blocksize of 4K for more realistic numbers

​

You need to install Prometheus.

Prometheus is more or less a time series database. https://prometheus.io/download/

Once the Prometheus server instance is up and running, you then tell the Ceph MGR instance where to publish the metrics to.

ceph mgr module enable prometheus ceph config-key set mgr/prometheus/server_addr $ipofprometheus ceph config-key set mgr/prometheus/server_port $portofprometheus

Then you can query your metrics from the prometheus server using Grafana, or whatever front end you like.

Thank for all of help. I'm successfully config CEPH Jewel Monitor by Grafana.

Fast deploy with docker prom/prometheus + digitalocean/ceph_exporter using this dashboard template https://grafana.com/dashboards/917

Hi @cryp7 So i will be presentation for you 2 case when you working with Journal Create with Journal #ceph-deploy osd create node1:sdc:/dev/sdb node2:sdc/dev/sdb node3:sdc:/dev/sdb This! is my script using for rebuild Journal SSD disk.

Hoping this can be help you.

Buffered I/O is enabled, I think it's been on from the beginning, as it's a rather fresh 16.2.x cluster.

What I read in several places is basically that bluestore has a very small deferred write buffer and will refuse to accept writes faster than the main storage media can write on average. I.e. even though I have bluestore WAL and DB on a ridiculously fast NVME drive, it won't write any faster than my SAS spinners can. That's a bummer 🙁

You can use M.2 to 5 sata adapters or 1x SATA to 5x SATA like this https://www.amazon.com/Bewinner-ADP6ST0-J05-Expansion-Converter-Motherboard/dp/B08545WGQM

Aliexpress have some of them

Not sure about performance with 1x sata to 5x, but m.2 adapter must be fine for sure.

Actually, you can use M.2 to 5 sata adapters or 1x SATA to 5x SATA like this https://www.amazon.com/Bewinner-ADP6ST0-J05-Expansion-Converter-Motherboard/dp/B08545WGQM

(aliexpress have some of them, for nice price)

If you don't need performance ofc.

More specifically, clock sync is essential to the PAXOS consensus algorithm that the mons use to elect a leader. The default warning threshold is 50ms; with Chrony (or even legacy ntpd) and judicious sync peers it's not difficult to maintain sub-ms accuracy.

plug: https://www.amazon.com/Learning-Ceph-scalable-reliable-solution/dp/1787127915