What are /r/ceph's favorite Products & Services?

From 3.5 billion Reddit comments

ceph-deploy can be pointed at local repos or you can configure through a proxy. I would probably copy the repo to the deploy server if you aren't going to have access to any other boxes.

you should have an odd number of monitors (3). You can use the deploy box imo, it won't do anything most of the time.

There are also very good ansible playbooks to consider instead of ceph-deploy. https://github.com/ceph/ceph-ansible

I would house rados gateways on separate hosts. Same for mds. mon/mgr are really the only recommended ok for colocation.

you should have one osd per disk, not per server. so if you have 30 4tb disks, you would have 30 osds on that host.

2 networks a public and a private is a recommended setup, its not necessary though.

ceph using the default settings is basically raid1 across a failure domain you specify (host, rack, etc).

A lot of good information is at http://ceph.com/docs/master/start

I don't know how well MDS is at this point. So I can't say anything about 1)

If you want to do the VM-approach use libvirt's built-in support for RBD rather than using the kernel rbd and mapping inside. That's what I would have done. (So 2. essentially)

http://ceph.com/docs/master/rbd/libvirt/

Bummer, but thank you and thank you u/ctrl_alt_lynx

I actually had really bad luck with ebay Mellanox or Intel cards and knock-offs.

For my TrueNAS machine, I ended up with a X520-10G-1S 10Gek card and it's served me really well.

I'm not sure if I see a real problem. Your tests are all with seemingly a large request size because troughput in MB/s is amazing and that would explain the low number of IOPs. Sequential and 'random' reads seems to bottleneck at the same MB/s so could that be networking? Check the interface of both the storage nodes and your client.

1500MB/s sounds like an odd number.

​

If you really want to test your cluster, use FIO for testing. https://github.com/axboe/fio

1. Create a RBD volume
2. Map the RBD volume on the RBD client (/dev/rbd0 maybe)
3. Use the raw mapped RBD device for FIO benchmarks, use randread/randwrite + blocksize of 4K for more realistic numbers

​

You need to install Prometheus.

Prometheus is more or less a time series database. https://prometheus.io/download/

Once the Prometheus server instance is up and running, you then tell the Ceph MGR instance where to publish the metrics to.

ceph mgr module enable prometheus ceph config-key set mgr/prometheus/server_addr $ipofprometheus ceph config-key set mgr/prometheus/server_port $portofprometheus

Then you can query your metrics from the prometheus server using Grafana, or whatever front end you like.

Thank for all of help. I'm successfully config CEPH Jewel Monitor by Grafana.

Fast deploy with docker prom/prometheus + digitalocean/ceph_exporter using this dashboard template https://grafana.com/dashboards/917

Hi @cryp7 So i will be presentation for you 2 case when you working with Journal Create with Journal #ceph-deploy osd create node1:sdc:/dev/sdb node2:sdc/dev/sdb node3:sdc:/dev/sdb This! is my script using for rebuild Journal SSD disk.

Hoping this can be help you.

Buffered I/O is enabled, I think it's been on from the beginning, as it's a rather fresh 16.2.x cluster.

What I read in several places is basically that bluestore has a very small deferred write buffer and will refuse to accept writes faster than the main storage media can write on average. I.e. even though I have bluestore WAL and DB on a ridiculously fast NVME drive, it won't write any faster than my SAS spinners can. That's a bummer 🙁

You can use M.2 to 5 sata adapters or 1x SATA to 5x SATA like this https://www.amazon.com/Bewinner-ADP6ST0-J05-Expansion-Converter-Motherboard/dp/B08545WGQM

Aliexpress have some of them

Not sure about performance with 1x sata to 5x, but m.2 adapter must be fine for sure.

Actually, you can use M.2 to 5 sata adapters or 1x SATA to 5x SATA like this https://www.amazon.com/Bewinner-ADP6ST0-J05-Expansion-Converter-Motherboard/dp/B08545WGQM

(aliexpress have some of them, for nice price)

If you don't need performance ofc.

More specifically, clock sync is essential to the PAXOS consensus algorithm that the mons use to elect a leader. The default warning threshold is 50ms; with Chrony (or even legacy ntpd) and judicious sync peers it's not difficult to maintain sub-ms accuracy.

plug: https://www.amazon.com/Learning-Ceph-scalable-reliable-solution/dp/1787127915