ctdb

package: samba-ctdb

title: ctdb(7)

summary: Clustered TDB

text:
# ctdb(7)

## NAME
ctdb - Clustered TDB

## DESCRIPTION
CTDB is a clustered database component in clustered Samba that provides a high-availability load-sharing CIFS server cluster.

The main functions of CTDB are:

•Provide a clustered version of the TDB database with automatic rebuild/recovery of the databases upon node failures.

•Monitor nodes in the cluster and services running on each node.

•Manage a pool of public IP addresses that are used to provide services to clients. Alternatively, CTDB can be used with LVS.

Combined with a cluster filesystem CTDB provides a full high-availablity (HA) environment for services such as clustered Samba, NFS and other services.

In addition to the CTDB manual pages there is much more information available at https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba.

## ANATOMY OF A CTDB CLUSTER
A CTDB cluster is a collection of nodes with 2 or more network interfaces. All nodes provide network (usually file/NAS) services to clients. Data served by file services is stored on shared storage (usually a cluster filesystem) that is accessible by all nodes.

CTDB provides an "all active" cluster, where services are load balanced across all nodes.

## CLUSTER LEADER
CTDB uses a cluster leader and follower model of cluster management. All nodes in a cluster elect one node to be the leader. The leader node coordinates privileged operations such as database recovery and IP address failover.

CTDB previously referred to the leader as the recovery master or recmaster. References to these terms may still be found in documentation and code.

## CLUSTER LOCK
CTDB uses a cluster lock to assert its privileged role in the cluster. This node takes the cluster lock when it becomes leader and holds the lock until it is no longer leader. The cluster lock helps CTDB to avoid a split brain, where a cluster becomes partitioned and each partition attempts to operate independently. Issues that can result from a split brain include file data corruption, because file locking metadata may not be tracked correctly.

CTDB previously referred to the cluster lock as the recovery lock. The abbreviation reclock is still used - just "clock" would be confusing.

CTDB is unable configure a default cluster lock, because this would depend on factors such as cluster filesystem mountpoints. However, running CTDB without a cluster lock is not recommended as there will be no split brain protection.

When a cluster lock is configured it is used as the election mechanism. Nodes race to take the cluster lock and the winner is the cluster leader. This avoids problems when a node wins an election but is unable to take the lock - this can occur if a cluster becomes partitioned (for example, due to a communication failure) and a different leader is elected by the nodes in each partition, or if the cluster filesystem has a high failover latency.

By default, the cluster lock is implemented using a file (specified by cluster lock in the [cluster] section of ctdb.conf(5)) residing in shared storage (usually) on a cluster filesystem. To support a cluster lock the cluster filesystem must support lock coherence. See ping_pong(1) for more details.

The cluster lock can also be implemented using an arbitrary cluster mutex helper (or call-out). This is indicated by using an exclamation point ('!') as the first character of the cluster lock parameter. For example, a value of !/usr/bin/myhelper cluster would run the given helper with the specified arguments. The helper will continue to run as long as it holds its mutex. See ctdb/doc/cluster_mutex_helper.txt in the source tree, and related code, for clues about writing helpers.

When a file is specified for the cluster lock parameter (i.e. no leading '!') the file lock is implemented by a default helper (/usr/lib/ctdb/ctdb_mutex_fcntl_helper). This helper has arguments as follows:

```
ctdb_mutex_fcntl_helper FILE [RECHECK-INTERVAL]

```

ctdb_mutex_fcntl_helper will take a lock on FILE and then check every RECHECK-INTERVAL seconds to ensure that FILE still exists and that its inode number is unchanged from when the lock was taken. The default value for RECHECK-INTERVAL is 5.

CTDB does sanity checks to ensure that the cluster lock is held as expected.

## PRIVATE VS PUBLIC ADDRESSES
Each node in a CTDB cluster has multiple IP addresses assigned to it:

•A single private IP address that is used for communication between nodes.

•One or more public IP addresses that are used to provide NAS or other services.

### Private address
Each node is configured with a unique, permanently assigned private address. This address is configured by the operating system. This address uniquely identifies a physical node in the cluster and is the address that CTDB daemons will use to communicate with the CTDB daemons on other nodes.

Private addresses are listed in the file /etc/ctdb/nodes). This file contains the list of private addresses for all nodes in the cluster, one per line. This file must be the same on all nodes in the cluster.

Some users like to put this configuration file in their cluster filesystem. A symbolic link or the nodes list parameter can be used.

Private addresses should not be used by clients to connect to services provided by the cluster.

It is strongly recommended that the private addresses are configured on a private network that is separate from client networks. This is because the CTDB protocol is both unauthenticated and unencrypted. If clients share the private network then steps need to be taken to stop injection of packets to relevant ports on the private addresses. It is also likely that CTDB protocol traffic between nodes could leak sensitive information if it can be intercepted.

Example /etc/ctdb/nodes for a four node cluster:

```
192.168.1.1
192.168.1.2
192.168.1.3
192.168.1.4

```

### Public addresses
Public addresses are used to provide services to clients. Public addresses are not configured at the operating system level and are not permanently associated with a particular node. Instead, they are managed by CTDB and are assigned to interfaces on physical nodes at runtime.

The CTDB cluster will assign/reassign these public addresses across the available healthy nodes in the cluster. When one node fails, its public addresses will be taken over by one or more other nodes in the cluster. This ensures that services provided by all public addresses are always available to clients, as long as there are nodes available capable of hosting this address.

The public address configuration is stored in /etc/ctdb/public_addresses on each node. This file contains a list of the public addresses that the node is capable of hosting, one per line. Each entry also contains the netmask and the interface to which the address should be assigned. If this file is missing then no public addresses are configured.

Some users who have the same public addresses on all nodes like to put this configuration file in their cluster filesystem. A symbolic link should be used in this case.

Example /etc/ctdb/public_addresses for a node that can host 4 public addresses, on 2 different interfaces:

```
10.1.1.1/24 eth1
10.1.1.2/24 eth1
10.1.2.1/24 eth2
10.1.2.2/24 eth2

```

In many cases the public addresses file will be the same on all nodes. However, it is possible to use different public address configurations on different nodes.

Example: 4 nodes partitioned into two subgroups:

```
Node 0:/etc/ctdb/public_addresses
	10.1.1.1/24 eth1
	10.1.1.2/24 eth1
Node 1:/etc/ctdb/public_addresses
	10.1.1.1/24 eth1
	10.1.1.2/24 eth1
Node 2:/etc/ctdb/public_addresses
	10.1.2.1/24 eth2
	10.1.2.2/24 eth2
Node 3:/etc/ctdb/public_addresses
	10.1.2.1/24 eth2
	10.1.2.2/24 eth2

```

In this example nodes 0 and 1 host two public addresses on the 10.1.1.x network while nodes 2 and 3 host two public addresses for the 10.1.2.x network.

Public address 10.1.1.1 can be hosted by either of nodes 0 or 1 and will be available to clients as long as at least one of these two nodes are available.

If both nodes 0 and 1 become unavailable then public address 10.1.1.1 also becomes unavailable. 10.1.1.1 can not be failed over to nodes 2 or 3 since these nodes do not have this public address configured.

The ctdb ip command can be used to view the current assignment of public addresses to physical nodes.

## NODE STATUS
The current status of each node in the cluster can be viewed by the ctdb status command.

A node can be in one of the following states:

OK

This node is healthy and fully functional. It hosts public addresses to provide services.

DISCONNECTED

This node is not reachable by other nodes via the private network. It is not currently participating in the cluster. It does not host public addresses to provide services. It might be shut down.

DISABLED

This node has been administratively disabled. This node is partially functional and participates in the cluster. However, it does not host public addresses to provide services.

UNHEALTHY

A service provided by this node has failed a health check and should be investigated. This node is partially functional and participates in the cluster. However, it does not host public addresses to provide services. Unhealthy nodes should be investigated and may require an administrative action to rectify.

BANNED

CTDB is not behaving as designed on this node. For example, it may have failed too many recovery attempts. Such nodes are banned from participating in the cluster for a configurable time period before they attempt to rejoin the cluster. A banned node does not host public addresses to provide services. All banned nodes should be investigated and may require an administrative action to rectify.

STOPPED

This node has been administratively exclude from the cluster. A stopped node does no participate in the cluster and does not host public addresses to provide services. This state can be used while performing maintenance on a node.

PARTIALLYONLINE

A node that is partially online participates in a cluster like a healthy (OK) node. Some interfaces to serve public addresses are down, but at least one interface is up. See also ctdb ifaces.

## CAPABILITIES
Cluster nodes can have several different capabilities enabled. These are listed below.

LEADER

Indicates that a node can become the CTDB cluster leader. The current leader is decided via an election held by all active nodes with this capability. Default is YES.

LMASTER

Indicates that a node can be the location master (LMASTER) for database records. The LMASTER always knows which node has the latest copy of a record in a volatile database. Default is YES.

The LEADER and LMASTER capabilities can be disabled when CTDB is used to create a cluster spanning across WAN links. In this case CTDB acts as a WAN accelerator.

## LVS
LVS is a mode where CTDB presents one single IP address for the entire cluster. This is an alternative to using public IP addresses and round-robin DNS to loadbalance clients across the cluster.

This is similar to using a layer-4 loadbalancing switch but with some restrictions.

One extra LVS public address is assigned on the public network to each LVS group. Each LVS group is a set of nodes in the cluster that presents the same LVS address public address to the outside world. Normally there would only be one LVS group spanning an entire cluster, but in situations where one CTDB cluster spans multiple physical sites it might be useful to have one LVS group for each site. There can be multiple LVS groups in a cluster but each node can only be member of one LVS group.

Client access to the cluster is load-balanced across the HEALTHY nodes in an LVS group. If no HEALTHY nodes exists then all nodes in the group are used, regardless of health status. CTDB will, however never load-balance LVS traffic to nodes that are BANNED, STOPPED, DISABLED or DISCONNECTED. The ctdb lvs command is used to show which nodes are currently load-balanced across.

In each LVS group, one of the nodes is selected by CTDB to be the LVS leader. This node receives all traffic from clients coming in to the LVS public address and multiplexes it across the internal network to one of the nodes that LVS is using. When responding to the client, that node will send the data back directly to the client, bypassing the LVS leader node. The command ctdb lvs leader will show which node is the current LVS leader.

The path used for a client I/O is:

1.Client sends request packet to LVS leader.

2.LVS leader passes the request on to one node across the internal network.

3.Selected node processes the request.

4.Node responds back to client.

This means that all incoming traffic to the cluster will pass through one physical node, which limits scalability. You can send more data to the LVS address that one physical node can multiplex. This means that you should not use LVS if your I/O pattern is write-intensive since you will be limited in the available network bandwidth that node can handle. LVS does work very well for read-intensive workloads where only smallish READ requests are going through the LVS leader bottleneck and the majority of the traffic volume (the data in the read replies) goes straight from the processing node back to the clients. For read-intensive i/o patterns you can achieve very high throughput rates in this mode.

Note: you can use LVS and public addresses at the same time.

If you use LVS, you must have a permanent address configured for the public interface on each node. This address must be routable and the cluster nodes must be configured so that all traffic back to client hosts are routed through this interface. This is also required in order to allow samba/winbind on the node to talk to the domain controller. This LVS IP address can not be used to initiate outgoing traffic.

Make sure that the domain controller and the clients are reachable from a node before you enable LVS. Also ensure that outgoing traffic to these hosts is routed out through the configured public interface.

### Configuration
To activate LVS on a CTDB node you must specify the CTDB_LVS_PUBLIC_IFACE, CTDB_LVS_PUBLIC_IP and CTDB_LVS_NODES configuration variables. CTDB_LVS_NODES specifies a file containing the private address of all nodes in the current node's LVS group.

Example:

```
CTDB_LVS_PUBLIC_IFACE=eth1
CTDB_LVS_PUBLIC_IP=10.1.1.237
CTDB_LVS_NODES=/etc/ctdb/lvs_nodes

```

Example /etc/ctdb/lvs_nodes:

```
192.168.1.2
192.168.1.3
192.168.1.4

```

Normally any node in an LVS group can act as the LVS leader. Nodes that are highly loaded due to other demands maybe flagged with the "follower-only" option in the CTDB_LVS_NODES file to limit the LVS functionality of those nodes.

LVS nodes file that excludes 192.168.1.4 from being the LVS leader node:

```
192.168.1.2
192.168.1.3
192.168.1.4 follower-only

```

## TRACKING AND RESETTING TCP CONNECTIONS
CTDB tracks TCP connections from clients to public IP addresses, on known ports. When an IP address moves from one node to another, all existing TCP connections to that IP address are reset. The node taking over this IP address will also send gratuitous ARPs (for IPv4, or neighbour advertisement, for IPv6). This allows clients to reconnect quickly, rather than waiting for TCP timeouts, which can be very long.

It is important that established TCP connections do not survive a release and take of a public IP address on the same node. Such connections can get out of sync with sequence and ACK numbers, potentially causing a disruptive ACK storm.

## NAT GATEWAY
NAT gateway (NATGW) is an optional feature that is used to configure fallback routing for nodes. This allows cluster nodes to connect to external services (e.g. DNS, AD, NIS and LDAP) when they do not host any public addresses (e.g. when they are unhealthy).

This also applies to node startup because CTDB marks nodes as UNHEALTHY until they have passed a "monitor" event. In this context, NAT gateway helps to avoid a "chicken and egg" situation where a node needs to access an external service to become healthy.

Another way of solving this type of problem is to assign an extra static IP address to a public interface on every node. This is simpler but it uses an extra IP address per node, while NAT gateway generally uses only one extra IP address.

### Operation
One extra NATGW public address is assigned on the public network to each NATGW group. Each NATGW group is a set of nodes in the cluster that shares the same NATGW address to talk to the outside world. Normally there would only be one NATGW group spanning an entire cluster, but in situations where one CTDB cluster spans multiple physical sites it might be useful to have one NATGW group for each site.

There can be multiple NATGW groups in a cluster but each node can only be member of one NATGW group.

In each NATGW group, one of the nodes is selected by CTDB to be the NATGW leader and the other nodes are consider to be NATGW followers. NATGW followers establish a fallback default route to the NATGW leader via the private network. When a NATGW follower hosts no public IP addresses then it will use this route for outbound connections. The NATGW leader hosts the NATGW public IP address and routes outgoing connections from follower nodes via this IP address. It also establishes a fallback default route.

### Configuration
NATGW is usually configured similar to the following example configuration:

```
CTDB_NATGW_NODES=/etc/ctdb/natgw_nodes
CTDB_NATGW_PRIVATE_NETWORK=192.168.1.0/24
CTDB_NATGW_PUBLIC_IP=10.0.0.227/24
CTDB_NATGW_PUBLIC_IFACE=eth0
CTDB_NATGW_DEFAULT_GATEWAY=10.0.0.1

```

Normally any node in a NATGW group can act as the NATGW leader. Some configurations may have special nodes that lack connectivity to a public network. In such cases, those nodes can be flagged with the "follower-only" option in the CTDB_NATGW_NODES file to limit the NATGW functionality of those nodes.

See the NAT GATEWAY section in ctdb-script.options(5) for more details of NATGW configuration.

### Implementation details
When the NATGW functionality is used, one of the nodes is selected to act as a NAT gateway for all the other nodes in the group when they need to communicate with the external services. The NATGW leader is selected to be a node that is most likely to have usable networks.

The NATGW leader hosts the NATGW public IP address CTDB_NATGW_PUBLIC_IP on the configured public interfaces CTDB_NATGW_PUBLIC_IFACE and acts as a router, masquerading outgoing connections from follower nodes via this IP address. If CTDB_NATGW_DEFAULT_GATEWAY is set then it also establishes a fallback default route to the configured this gateway with a metric of 10. A metric 10 route is used so it can co-exist with other default routes that may be available.

A NATGW follower establishes its fallback default route to the NATGW leader via the private network CTDB_NATGW_PRIVATE_NETWORKwith a metric of 10. This route is used for outbound connections when no other default route is available because the node hosts no public addresses. A metric 10 routes is used so that it can co-exist with other default routes that may be available when the node is hosting public addresses.

CTDB_NATGW_STATIC_ROUTES can be used to have NATGW create more specific routes instead of just default routes.

This is implemented in the 11.natgw eventscript. Please see the eventscript file and the NAT GATEWAY section in ctdb-script.options(5) for more details.

## POLICY ROUTING
Policy routing is an optional CTDB feature to support complex network topologies. Public addresses may be spread across several different networks (or VLANs) and it may not be possible to route packets from these public addresses via the system's default route. Therefore, CTDB has support for policy routing via the 13.per_ip_routing eventscript. This allows routing to be specified for packets sourced from each public address. The routes are added and removed as CTDB moves public addresses between nodes.

### Configuration variables
There are 4 configuration variables related to policy routing: CTDB_PER_IP_ROUTING_CONF, CTDB_PER_IP_ROUTING_RULE_PREF, CTDB_PER_IP_ROUTING_TABLE_ID_LOW, CTDB_PER_IP_ROUTING_TABLE_ID_HIGH. See the POLICY ROUTING section in ctdb-script.options(5) for more details.

### Configuration
The format of each line of CTDB_PER_IP_ROUTING_CONF is:

```
<public_address> <network> [ <gateway> ]

```

Leading whitespace is ignored and arbitrary whitespace may be used as a separator. Lines that have a "public address" item that doesn't match an actual public address are ignored. This means that comment lines can be added using a leading character such as '#', since this will never match an IP address.

A line without a gateway indicates a link local route.

For example, consider the configuration line:

```
  192.168.1.99	192.168.1.0/24

```

If the corresponding public_addresses line is:

```
  192.168.1.99/24     eth2,eth3

```

CTDB_PER_IP_ROUTING_RULE_PREF is 100, and CTDB adds the address to eth2 then the following routing information is added:

```
  ip rule add from 192.168.1.99 pref 100 table ctdb.192.168.1.99

  ip route add 192.168.1.0/24 dev eth2 table ctdb.192.168.1.99

```

This causes traffic from 192.168.1.99 to 192.168.1.0/24 go via eth2.

The ip rule command will show (something like - depending on other public addresses and other routes on the system):

```
  0:		from all lookup local

  100:		from 192.168.1.99 lookup ctdb.192.168.1.99

  32766:	from all lookup main

  32767:	from all lookup default

```

ip route show table ctdb.192.168.1.99 will show:

```
  192.168.1.0/24 dev eth2 scope link

```

The usual use for a line containing a gateway is to add a default route corresponding to a particular source address. Consider this line of configuration:

```
  192.168.1.99	0.0.0.0/0	192.168.1.1

```

In the situation described above this will cause an extra routing command to be executed:

```
  ip route add 0.0.0.0/0 via 192.168.1.1 dev eth2 table ctdb.192.168.1.99

```

With both configuration lines, ip route show table ctdb.192.168.1.99 will show:

```
  192.168.1.0/24 dev eth2 scope link

  default via 192.168.1.1 dev eth2

```

### Sample configuration
Here is a more complete example configuration.

```
/etc/ctdb/public_addresses:

  192.168.1.98	eth2,eth3

  192.168.1.99	eth2,eth3
/etc/ctdb/policy_routing:

  192.168.1.98 192.168.1.0/24

  192.168.1.98 192.168.200.0/24	192.168.1.254

  192.168.1.98 0.0.0.0/0 	192.168.1.1

  192.168.1.99 192.168.1.0/24

  192.168.1.99 192.168.200.0/24	192.168.1.254

  192.168.1.99 0.0.0.0/0 	192.168.1.1

```

The routes local packets as expected, the default route is as previously discussed, but packets to 192.168.200.0/24 are routed via the alternate gateway 192.168.1.254.

## NOTIFICATIONS
When certain state changes occur in CTDB, it can be configured to perform arbitrary actions via notifications. For example, sending SNMP traps or emails when a node becomes unhealthy or similar.

The notification mechanism runs all executable files ending in ".script" in /etc/ctdb/events/notification/, ignoring any failures and continuing to run all files.

CTDB currently generates notifications after CTDB changes to these states:

init

setup

startup

healthy

unhealthy

## LOG LEVELS
Valid log levels, in increasing order of verbosity, are:

ERROR

WARNING

NOTICE

INFO

DEBUG

## REMOTE CLUSTER NODES
It is possible to have a CTDB cluster that spans across a WAN link. For example where you have a CTDB cluster in your datacentre but you also want to have one additional CTDB node located at a remote branch site. This is similar to how a WAN accelerator works but with the difference that while a WAN-accelerator often acts as a Proxy or a MitM, in the ctdb remote cluster node configuration the Samba instance at the remote site IS the genuine server, not a proxy and not a MitM, and thus provides 100% correct CIFS semantics to clients.

See the cluster as one single multihomed samba server where one of the NICs (the remote node) is very far away.

NOTE: This does require that the cluster filesystem you use can cope with WAN-link latencies. Not all cluster filesystems can handle WAN-link latencies! Whether this will provide very good WAN-accelerator performance or it will perform very poorly depends entirely on how optimized your cluster filesystem is in handling high latency for data and metadata operations.

To activate a node as being a remote cluster node you need to set the following two parameters in /etc/ctdb/ctdb.conf for the remote node:

```
[legacy]

  lmaster capability = false

  leader capability = false

```

Verify with the command "ctdb getcapabilities" that that node no longer has the leader or the lmaster capabilities.

## SEE ALSO
ctdb(1), ctdbd(1), ctdb_diagnostics(1), ltdbtool(1), onnode(1), ping_pong(1), ctdb.conf(5), ctdb-script.options(5), ctdb.sysconfig(5), ctdb-statistics(7), ctdb-tunables(7), https://wiki.samba.org/index.php/CTDB_and_Clustered_Samba, http://ctdb.samba.org/

## AUTHOR
This documentation was written by Ronnie Sahlberg, Amitay Isaacs, Martin Schwenke

## COPYRIGHT
Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses.


ctdb-statistics

title: ctdb-statistics(7)

summary: CTDB statistics output

text:
# ctdb-statistics(7)

## NAME
ctdb-statistics - CTDB statistics output

## OVERALL STATISTICS
CTDB maintains information about various messages communicated and some of the important operations per node. See the ctdb(1) commands statistics and statisticsreset for displaying statistics.

### Example: ctdb statistics

```
CTDB version 1
Current time of statistics  :                Fri Sep 12 13:32:32 2014
Statistics collected since  : (000 01:49:20) Fri Sep 12 11:43:12 2014

 num_clients                        6

 frozen                             0

 recovering                         0

 num_recoveries                     2

 client_packets_sent           281293

 client_packets_recv           296317

 node_packets_sent             452387

 node_packets_recv             182394

 keepalive_packets_sent          3927

 keepalive_packets_recv          3928

 node

     req_call                   48605

     reply_call                     1

     req_dmaster                23404

     reply_dmaster              24917

     reply_error                    0

     req_message                  958

     req_control               197513

     reply_control             153705

 client

     req_call                  130866

     req_message                  770

     req_control               168921

 timeouts

     call                           0

     control                        0

     traverse                       0

 locks

     num_calls                    220

     num_current                    0

     num_pending                    0

     num_failed                     0

 total_calls                   130866

 pending_calls                      0

 childwrite_calls                   1

 pending_childwrite_calls             0

 memory_used                   334490

 max_hop_count                     18

 total_ro_delegations               2

 total_ro_revokes                   2

 hop_count_buckets: 42816 5464 26 1 0 0 0 0 0 0 0 0 0 0 0 0

 lock_buckets: 9 165 14 15 7 2 2 0 0 0 0 0 0 0 0 0

 locks_latency      MIN/AVG/MAX     0.000685/0.160302/6.369342 sec out of 214

 reclock_ctdbd      MIN/AVG/MAX     0.004940/0.004969/0.004998 sec out of 2

 reclock_recd       MIN/AVG/MAX     0.000000/0.000000/0.000000 sec out of 0

 call_latency       MIN/AVG/MAX     0.000006/0.000719/4.562991 sec out of 126626

 childwrite_latency MIN/AVG/MAX     0.014527/0.014527/0.014527 sec out of 1

```

### CTDB version
Version of the ctdb protocol used by the node.

### Current time of statistics
Time when the statistics are generated.

This is useful when collecting statistics output periodically for post-processing.

### Statistics collected since
Time when ctdb was started or the last time statistics was reset. The output shows the duration and the timestamp.

### num_clients
Number of processes currently connected to CTDB's unix socket. This includes recovery daemon, ctdb tool and samba processes (smbd, winbindd).

### frozen
1 if the databases are currently frozen, 0 otherwise.

### recovering
1 if recovery is active, 0 otherwise.

### num_recoveries
Number of recoveries since the start of ctdb or since the last statistics reset.

### client_packets_sent
Number of packets sent to client processes via unix domain socket.

### client_packets_recv
Number of packets received from client processes via unix domain socket.

### node_packets_sent
Number of packets sent to the other nodes in the cluster via TCP.

### node_packets_recv
Number of packets received from the other nodes in the cluster via TCP.

### keepalive_packets_sent
Number of keepalive messages sent to other nodes.

CTDB periodically sends keepalive messages to other nodes. See KeepaliveInterval tunable in ctdb-tunables(7) for more details.

### keepalive_packets_recv
Number of keepalive messages received from other nodes.

### node
This section lists various types of messages processed which originated from other nodes via TCP.

req_call

Number of REQ_CALL messages from the other nodes.

reply_call

Number of REPLY_CALL messages from the other nodes.

req_dmaster

Number of REQ_DMASTER messages from the other nodes.

reply_dmaster

Number of REPLY_DMASTER messages from the other nodes.

reply_error

Number of REPLY_ERROR messages from the other nodes.

req_message

Number of REQ_MESSAGE messages from the other nodes.

req_control

Number of REQ_CONTROL messages from the other nodes.

reply_control

Number of REPLY_CONTROL messages from the other nodes.

req_tunnel

Number of REQ_TUNNEL messages from the other nodes.

### client
This section lists various types of messages processed which originated from clients via unix domain socket.

req_call

Number of REQ_CALL messages from the clients.

req_message

Number of REQ_MESSAGE messages from the clients.

req_control

Number of REQ_CONTROL messages from the clients.

req_tunnel

Number of REQ_TUNNEL messages from the clients.

### timeouts
This section lists timeouts occurred when sending various messages.

call

Number of timeouts for REQ_CALL messages.

control

Number of timeouts for REQ_CONTROL messages.

traverse

Number of timeouts for database traverse operations.

### locks
This section lists locking statistics.

num_calls

Number of completed lock calls. This includes database locks and record locks.

num_current

Number of scheduled lock calls. This includes database locks and record locks.

num_pending

Number of queued lock calls. This includes database locks and record locks.

num_failed

Number of failed lock calls. This includes database locks and record locks.

### total_calls
Number of req_call messages processed from clients. This number should be same as client --> req_call.

### pending_calls
Number of req_call messages which are currently being processed. This number indicates the number of record migrations in flight.

### childwrite_calls
Number of record update calls. Record update calls are used to update a record under a transaction.

### pending_childwrite_calls
Number of record update calls currently active.

### memory_used
The amount of memory in bytes currently used by CTDB using talloc. This includes all the memory used for CTDB's internal data structures. This does not include the memory mapped TDB databases.

### max_hop_count
The maximum number of hops required for a record migration request to obtain the record. High numbers indicate record contention.

### total_ro_delegations
Number of readonly delegations created.

### total_ro_revokes
Number of readonly delegations that were revoked. The difference between total_ro_revokes and total_ro_delegations gives the number of currently active readonly delegations.

### hop_count_buckets
Distribution of migration requests based on hop counts values. Buckets are 0, < 2, < 4, < 8, < 16, < 32, < 64, < 128, < 256, < 512, < 1024, < 2048, < 4096, < 8192, < 16384, ≥ 16384.

### lock_buckets
Distribution of record lock requests based on time required to obtain locks. Buckets are < 1ms, < 10ms, < 100ms, < 1s, < 2s, < 4s, < 8s, < 16s, < 32s, < 64s, ≥ 64s.

### locks_latency
The minimum, the average and the maximum time (in seconds) required to obtain record locks.

### reclock_ctdbd
The minimum, the average and the maximum time (in seconds) required to check if recovery lock is still held by recovery daemon when recovery mode is changed. This check is done in ctdb daemon.

### reclock_recd
The minimum, the average and the maximum time (in seconds) required to check if recovery lock is still held by recovery daemon during recovery. This check is done in recovery daemon.

### call_latency
The minimum, the average and the maximum time (in seconds) required to process a REQ_CALL message from client. This includes the time required to migrate a record from remote node, if the record is not available on the local node.

### childwrite_latency
Default: 0

The minimum, the average and the maximum time (in seconds) required to update records under a transaction.

## DATABASE STATISTICS
CTDB maintains per database statistics about important operations. See the ctdb(1) command dbstatistics for displaying database statistics.

### Example: ctdb dbstatistics notify_index.tdb

```
DB Statistics: notify_index.tdb

 ro_delegations                     0

 ro_revokes                         0

 locks

     total                        131

     failed                         0

     current                        0

     pending                        0

 hop_count_buckets: 9890 5454 26 1 0 0 0 0 0 0 0 0 0 0 0 0

 lock_buckets: 4 117 10 0 0 0 0 0 0 0 0 0 0 0 0 0

 locks_latency      MIN/AVG/MAX     0.000683/0.004198/0.014730 sec out of 131

 Num Hot Keys:     3

     Count:7 Key:2f636c75737465726673

     Count:18 Key:2f636c757374657266732f64617461

     Count:7 Key:2f636c757374657266732f646174612f636c69656e7473

```

### DB Statistics
Name of the database.

### ro_delegations
Number of readonly delegations created in the database.

### ro_revokes
Number of readonly delegations revoked. The difference in ro_delegations and ro_revokes indicates the currently active readonly delegations.

### locks
This section lists locking statistics.

total

Number of completed lock calls. This includes database locks and record locks.

failed

Number of failed lock calls. This includes database locks and record locks.

current

Number of scheduled lock calls. This includes database locks and record locks.

pending

Number of queued lock calls. This includes database locks and record locks.

### hop_count_buckets
Distribution of migration requests based on hop counts values. Buckets are 0, < 2, < 4, < 8, < 16, < 32, < 64, < 128, < 256, < 512, < 1024, < 2048, < 4096, < 8192, < 16384, ≥ 16384.

### lock_buckets
Distribution of record lock requests based on time required to obtain locks. Buckets are < 1ms, < 10ms, < 100ms, < 1s, < 2s, < 4s, < 8s, < 16s, < 32s, < 64s, ≥ 64s.

### locks_latency
The minimum, the average and the maximum time (in seconds) required to obtain record locks.

### Num Hot Keys
Number of contended records determined by hop count. CTDB keeps track of top 10 hot records and the output shows hex encoded keys for the hot records.

## SEE ALSO
ctdb(1), ctdbd(1), ctdb-tunables(7), http://ctdb.samba.org/

## AUTHOR
This documentation was written by Amitay Isaacs, Martin Schwenke

## COPYRIGHT
Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses.

branch: sisyphus


ctdb-tunables

title: ctdb-tunables(7)

summary: CTDB tunable configuration variables

text:
# ctdb-tunables(7)

## NAME
ctdb-tunables - CTDB tunable configuration variables

## DESCRIPTION
CTDB's behaviour can be configured by setting run-time tunable variables. This lists and describes all tunables. See the ctdb(1) listvars, setvar and getvar commands for more details.

Unless otherwise stated, tunables should be set to the same value on all nodes. Setting tunables to different values across nodes may produce unexpected results. Future releases may set (some or most) tunables globally across the cluster but doing so is currently a manual process.

Tunables can be set at startup from the /etc/ctdb/ctdb.tunables configuration file.

```
TUNABLE=VALUE

```

Comment lines beginning with '#' are permitted. Whitespace may be used for formatting/alignment. VALUE must be a non-negative integer and must be the last thing on a line (i.e. no trailing garbage, trailing comments are not permitted).

For example:

```
MonitorInterval=20

```

The available tunable variables are listed alphabetically below.

### AllowClientDBAttach
Default: 1

When set to 0, clients are not allowed to attach to any databases. This can be used to temporarily block any new processes from attaching to and accessing the databases. This is mainly used for detaching a volatile database using 'ctdb detach'.

### AllowMixedVersions
Default: 0

CTDB will not allow incompatible versions to co-exist in a cluster. If a version mismatch is found, then losing CTDB will shutdown. To disable the incompatible version check, set this tunable to 1.

For version checking, CTDB uses major and minor version. For example, CTDB 4.6.1 and CTDB 4.6.2 are matching versions; CTDB 4.5.x and CTDB 4.6.y do not match.

CTDB with version check support will lose to CTDB without version check support. Between two different CTDB versions with version check support, one running for less time will lose. If the running time for both CTDB versions with version check support is equal (to seconds), then the older version will lose. The losing CTDB daemon will shutdown.

### AllowUnhealthyDBRead
Default: 0

When set to 1, ctdb allows database traverses to read unhealthy databases. By default, ctdb does not allow reading records from unhealthy databases.

### ControlTimeout
Default: 60

This is the default setting for timeout for when sending a control message to either the local or a remote ctdb daemon.

### DatabaseHashSize
Default: 100001

Number of the hash chains for the local store of the tdbs that ctdb manages.

### DatabaseMaxDead
Default: 5

Maximum number of dead records per hash chain for the tdb databases managed by ctdb.

### DBRecordCountWarn
Default: 100000

When set to non-zero, ctdb will log a warning during recovery if a database has more than this many records. This will produce a warning if a database grows uncontrollably with orphaned records.

### DBRecordSizeWarn
Default: 10000000

When set to non-zero, ctdb will log a warning during recovery if a single record is bigger than this size. This will produce a warning if a database record grows uncontrollably.

### DBSizeWarn
Default: 1000000000

When set to non-zero, ctdb will log a warning during recovery if a database size is bigger than this. This will produce a warning if a database grows uncontrollably.

### DeferredAttachTO
Default: 120

When databases are frozen we do not allow clients to attach to the databases. Instead of returning an error immediately to the client, the attach request from the client is deferred until the database becomes available again at which stage we respond to the client.

This timeout controls how long we will defer the request from the client before timing it out and returning an error to the client.

### ElectionTimeout
Default: 3

The number of seconds to wait for the election of recovery master to complete. If the election is not completed during this interval, then that round of election fails and ctdb starts a new election.

### EnableBans
Default: 1

This parameter allows ctdb to ban a node if the node is misbehaving.

When set to 0, this disables banning completely in the cluster and thus nodes can not get banned, even it they break. Don't set to 0 unless you know what you are doing.

### EventScriptTimeout
Default: 30

Maximum time in seconds to allow an event to run before timing out. This is the total time for all enabled scripts that are run for an event, not just a single event script.

Note that timeouts are ignored for some events ("takeip", "releaseip", "startrecovery", "recovered") and converted to success. The logic here is that the callers of these events implement their own additional timeout.

### FetchCollapse
Default: 1

This parameter is used to avoid multiple migration requests for the same record from a single node. All the record requests for the same record are queued up and processed when the record is migrated to the current node.

When many clients across many nodes try to access the same record at the same time this can lead to a fetch storm where the record becomes very active and bounces between nodes very fast. This leads to high CPU utilization of the ctdbd daemon, trying to bounce that record around very fast, and poor performance. This can improve performance and reduce CPU utilization for certain workloads.

### HopcountMakeSticky
Default: 50

For database(s) marked STICKY (using 'ctdb setdbsticky'), any record that is migrating so fast that hopcount exceeds this limit is marked as STICKY record for StickyDuration seconds. This means that after each migration the sticky record will be kept on the node StickyPindownmilliseconds and prevented from being migrated off the node.

This will improve performance for certain workloads, such as locking.tdb if many clients are opening/closing the same file concurrently.

### IPAllocAlgorithm
Default: 2

Selects the algorithm that CTDB should use when doing public IP address allocation. Meaningful values are:

0

Deterministic IP address allocation. This is a simple and fast option. However, it can cause unnecessary address movement during fail-over because each address has a "home" node. Works badly when some nodes do not have any addresses defined. Should be used with care when addresses are defined across multiple networks.

You can override the automatic "home" node allocation by creating a file "home_nodes" next to the "public_addresses" file. As an example the following "home_nodes" file assigns the address 192.168.1.1 to node 0 and 192.168.1.2 to node 2:

```
	      192.168.1.1 0
	      192.168.1.2 2

```

1

Non-deterministic IP address allocation. This is a relatively fast option that attempts to do a minimise unnecessary address movements. Addresses do not have a "home" node. Rebalancing is limited but it usually adequate. Works badly when addresses are defined across multiple networks.

2

LCP2 IP address allocation. Uses a heuristic to assign addresses defined across multiple networks, usually balancing addresses on each network evenly across nodes. Addresses do not have a "home" node. Minimises unnecessary address movements. The algorithm is complex, so is slower than other choices for a large number of addresses. However, it can calculate an optimal assignment of 900 addresses in under 10 seconds on modern hardware.

If the specified value is not one of these then the default will be used.

### KeepaliveInterval
Default: 5

How often in seconds should the nodes send keep-alive packets to each other.

### KeepaliveLimit
Default: 5

After how many keepalive intervals without any traffic should a node wait until marking the peer as DISCONNECTED.

If a node has hung, it can take KeepaliveInterval * (KeepaliveLimit + 1) seconds before ctdb determines that the node is DISCONNECTED and performs a recovery. This limit should not be set too high to enable early detection and avoid any application timeouts (e.g. SMB1) to kick in before the fail over is completed.

### LockProcessesPerDB
Default: 200

This is the maximum number of lock helper processes ctdb will create for obtaining record locks. When ctdb cannot get a record lock without blocking, it creates a helper process that waits for the lock to be obtained.

### LogLatencyMs
Default: 0

When set to non-zero, ctdb will log if certains operations take longer than this value, in milliseconds, to complete. These operations include "process a record request from client", "take a record or database lock", "update a persistent database record" and "vacuum a database".

### MaxQueueDropMsg
Default: 1000000

This is the maximum number of messages to be queued up for a client before ctdb will treat the client as hung and will terminate the client connection.

### MonitorInterval
Default: 15

How often should ctdb run the 'monitor' event in seconds to check for a node's health.

### MonitorTimeoutCount
Default: 20

How many 'monitor' events in a row need to timeout before a node is flagged as UNHEALTHY. This setting is useful if scripts can not be written so that they do not hang for benign reasons.

### NoIPFailback
Default: 0

When set to 1, ctdb will not perform failback of IP addresses when a node becomes healthy. When a node becomes UNHEALTHY, ctdb WILL perform failover of public IP addresses, but when the node becomes HEALTHY again, ctdb will not fail the addresses back.

Use with caution! Normally when a node becomes available to the cluster ctdb will try to reassign public IP addresses onto the new node as a way to distribute the workload evenly across the clusternode. Ctdb tries to make sure that all running nodes have approximately the same number of public addresses it hosts.

When you enable this tunable, ctdb will no longer attempt to rebalance the cluster by failing IP addresses back to the new nodes. An unbalanced cluster will therefore remain unbalanced until there is manual intervention from the administrator. When this parameter is set, you can manually fail public IP addresses over to the new node(s) using the 'ctdb moveip' command.

### NoIPTakeover
Default: 0

When set to 1, ctdb will not allow IP addresses to be failed over to other nodes. Any IP addresses already hosted on healthy nodes will remain. Any IP addresses hosted on unhealthy nodes will be released by unhealthy nodes and will become un-hosted.

### PullDBPreallocation
Default: 10*1024*1024

This is the size of a record buffer to pre-allocate for sending reply to PULLDB control. Usually record buffer starts with size of the first record and gets reallocated every time a new record is added to the record buffer. For a large number of records, this can be very inefficient to grow the record buffer one record at a time.

### QueueBufferSize
Default: 1024

This is the maximum amount of data (in bytes) ctdb will read from a socket at a time.

For a busy setup, if ctdb is not able to process the TCP sockets fast enough (large amount of data in Recv-Q for tcp sockets), then this tunable value should be increased. However, large values can keep ctdb busy processing packets and prevent ctdb from handling other events.

### RecBufferSizeLimit
Default: 1000000

This is the limit on the size of the record buffer to be sent in various controls. This limit is used by new controls used for recovery and controls used in vacuuming.

### RecdFailCount
Default: 10

If the recovery daemon has failed to ping the main daemon for this many consecutive intervals, the main daemon will consider the recovery daemon as hung and will try to restart it to recover.

### RecdPingTimeout
Default: 60

If the main daemon has not heard a "ping" from the recovery daemon for this many seconds, the main daemon will log a message that the recovery daemon is potentially hung. This also increments a counter which is checked against RecdFailCount for detection of hung recovery daemon.

### RecLockLatencyMs
Default: 1000

When using a reclock file for split brain prevention, if set to non-zero this tunable will make the recovery daemon log a message if the fcntl() call to lock/testlock the recovery file takes longer than this number of milliseconds.

### RecoverInterval
Default: 1

How frequently in seconds should the recovery daemon perform the consistency checks to determine if it should perform a recovery.

### RecoverTimeout
Default: 120

This is the default setting for timeouts for controls when sent from the recovery daemon. We allow longer control timeouts from the recovery daemon than from normal use since the recovery daemon often use controls that can take a lot longer than normal controls.

### RecoveryBanPeriod
Default: 300

The duration in seconds for which a node is banned if the node fails during recovery. After this time has elapsed the node will automatically get unbanned and will attempt to rejoin the cluster.

A node usually gets banned due to real problems with the node. Don't set this value too small. Otherwise, a problematic node will try to re-join cluster too soon causing unnecessary recoveries.

### RecoveryDropAllIPs
Default: 120

If a node is stuck in recovery, or stopped, or banned, for this many seconds, then ctdb will release all public addresses on that node.

### RecoveryGracePeriod
Default: 120

During recoveries, if a node has not caused recovery failures during the last grace period in seconds, any records of transgressions that the node has caused recovery failures will be forgiven. This resets the ban-counter back to zero for that node.

### RepackLimit
Default: 10000

During vacuuming, if the number of freelist records are more than RepackLimit, then the database is repacked to get rid of the freelist records to avoid fragmentation.

### RerecoveryTimeout
Default: 10

Once a recovery has completed, no additional recoveries are permitted until this timeout in seconds has expired.

### SeqnumInterval
Default: 1000

Some databases have seqnum tracking enabled, so that samba will be able to detect asynchronously when there has been updates to the database. Every time a database is updated its sequence number is increased.

This tunable is used to specify in milliseconds how frequently ctdb will send out updates to remote nodes to inform them that the sequence number is increased.

### StatHistoryInterval
Default: 1

Granularity of the statistics collected in the statistics history. This is reported by 'ctdb stats' command.

### StickyDuration
Default: 600

Once a record has been marked STICKY, this is the duration in seconds, the record will be flagged as a STICKY record.

### StickyPindown
Default: 200

Once a STICKY record has been migrated onto a node, it will be pinned down on that node for this number of milliseconds. Any request from other nodes to migrate the record off the node will be deferred.

### TakeoverTimeout
Default: 9

This is the duration in seconds in which ctdb tries to complete IP failover.

### TickleUpdateInterval
Default: 20

Every TickleUpdateInterval seconds, ctdb synchronizes the client connection information across nodes.

### TraverseTimeout
Default: 20

This is the duration in seconds for which a database traverse is allowed to run. If the traverse does not complete during this interval, ctdb will abort the traverse.

### VacuumFastPathCount
Default: 60

During a vacuuming run, ctdb usually processes only the records marked for deletion also called the fast path vacuuming. After finishing VacuumFastPathCount number of fast path vacuuming runs, ctdb will trigger a scan of complete database for any empty records that need to be deleted.

### VacuumInterval
Default: 10

Periodic interval in seconds when vacuuming is triggered for volatile databases.

### VacuumMaxRunTime
Default: 120

The maximum time in seconds for which the vacuuming process is allowed to run. If vacuuming process takes longer than this value, then the vacuuming process is terminated.

### VerboseMemoryNames
Default: 0

When set to non-zero, ctdb assigns verbose names for some of the talloc allocated memory objects. These names are visible in the talloc memory report generated by 'ctdb dumpmemory'.

## FILES>
/etc/ctdb/ctdb.tunables

## SEE ALSO
ctdb(1), ctdbd(1), ctdb.conf(5), ctdb(7), http://ctdb.samba.org/

## AUTHOR
This documentation was written by Ronnie Sahlberg, Amitay Isaacs, Martin Schwenke

## COPYRIGHT
Copyright © 2007 Andrew Tridgell, Ronnie Sahlberg

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses.


package: samba-ctdb-ceph-mutex

title: ctdb_mutex_ceph_rados_helper(7)

summary: Ceph RADOS cluster mutex helper

text:
# ctdb_mutex_ceph_rados_helper(7)

## NAME
ctdb_mutex_ceph_rados_helper - Ceph RADOS cluster mutex helper

## DESCRIPTION
ctdb_mutex_ceph_rados_helper can be used as a cluster lock provider for CTDB. When configured, split brain avoidance during CTDB recovery will be handled using locks against an object located in a Ceph RADOS pool. To enable this functionality, include the following line in the [cluster] section of ctdb.conf(5):

```
cluster lock = !ctdb_mutex_ceph_rados_helper [Cluster] [User] [Pool] [Object] [Timeout] [-n Namespace]
Cluster: Ceph cluster name (e.g. ceph)
User: Ceph cluster user name (e.g. client.admin)
Pool: Ceph RADOS pool name
Object: Ceph RADOS object name
Timeout: Ceph RADOS lock duration in seconds (optional)
Namespace: Ceph RADOS pool namespace (optional)

```

The Ceph cluster Cluster must be up and running, with a configuration, and keyring file for User located in a librados default search path (e.g. /etc/ceph/). Pool must already exist.

For informational purposes, ctdb_mutex_ceph_rados_helper will also register the cluster lock holder in Ceph Manager's service map.

## SEE ALSO
ctdb(7), ctdbd(1), http://ctdb.samba.org/

## AUTHOR
This documentation was written by David Disseldorp

## COPYRIGHT
Copyright © 2016 David Disseldorp

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses.


ctdb-etcd

package: samba-ctdb-etcd-mutex

title: ctdb-etcd(7)

summary: CTDB etcd integration

text:
# ctdb-etcd(7)

## NAME
ctdb-etcd - CTDB etcd integration

## SYNOPSIS
ctdb_etcd_lock

## DESCRIPTION
ctdb_etcd_lock is intended to be run as a mutex helper for CTDB. It will try to connect to an existing etcd cluster and grab a lock in that cluster to function as CTDB's cluster lock. Please see ctdb/doc/cluster_mutex_helper.txt for details on the mutex helper API. To use this, include the following line in the [cluster] section of ctdb.conf(5):

```
cluster lock = !/usr/local/usr/libexec/ctdb/ctdb_etcd_lock

```

You can also pass "-v", "-vv", or "-vvv" to include verbose output in the CTDB log. Additional "v"s indicate increases in verbosity.

This mutex helper expects the system Python interpreter to have access to the etcd Python module. It also expects an etcd cluster to be configured and running. To integrate with this, there is an optional config file of the following format:

```
key = value

```

The following configuration parameters (and their defaults) are defined for use by ctdb_etcd_lock:

```
port      = 2379   # connecting port for the etcd cluster
lock_ttl  = 9      # seconds for TTL
refresh   = 2      # seconds between attempts to maintain lock
locks_dir = _ctdb  # where to store CTDB locks in etcd

                   # The final etcd directory for any given lock looks like:

                   #   /_locks/{locks_dir}/{netbios name}/

```

In addition, any keyword parameter that can be used to configure an etcd client may be specified and modified here. For more documentation on these parameters, see here: https://github.com/jplana/python-etcd/

## SEE ALSO
ctdb(7), ctdbd(1), http://ctdb.samba.org/

## AUTHOR
This documentation was written by Jose A. Rivera

## COPYRIGHT
Copyright © 2016 Jose A. Rivera

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, see http://www.gnu.org/licenses.


package: samba-test

title: traffic_learner(7)

text:
# traffic_learner(7)

## NAME
traffic_learner - Samba tool to assist with traffic generation.

## SYNOPSIS
traffic_learner {-o OUTPUT_FILE ...} [-h] [--dns-mode {inline|count}] [SUMMARY_FILE] [SUMMARY_FILE ...]

## DESCRIPTION
This tool is part of the samba(7) suite.

This tool assists with generation of Samba traffic. It takes a traffic-summary file (produced by traffic_summary.pl) as input and produces a traffic-model file that can be used by traffic_replay for traffic generation.

The model file summarizes the types of traffic ('conversations' between a host and a Samba DC) that occur on a network. The model file describes the traffic in a way that allows it to be scaled so that either more (or fewer) packets get sent, and the packets can be sent at a faster (or slower) rate than that seen in the network.

## OPTIONS
-h|--help

Print a summary of command line options.

SUMMARY_FILE

File containing a network traffic-summary. The traffic-summary file should be generated by traffic_summary.pl from a packet capture of actual network traffic. More than one file can be specified, in which case the traffic will be combined into a single traffic-model. If no SUMMARY_FILE is specified, this tool will read the traffic-summary from STDIN, i.e. you can pipe the output from traffic_summary.pl directly to this tool.

-o|--out OUTPUT_FILE

The traffic-model that is produced will be written to this file. The OUTPUT_FILE can then be passed to traffic_replay to generate (and manipulate) Samba network traffic.

--dns-mode [inline|count]

How DNS traffic should be handled by the model.

## EXAMPLES
To take a traffic-summary file and produce a traffic-model file, use:

traffic_learner traffic-summary.txt -o traffic-model.txt

To generate a traffic-model from a packet capture, you can pipe the traffic summary to STDIN using:

tshark -r capture.pcapng -T pdml | traffic_summary.pl | traffic_learner -o traffic-model.txt

## OUTPUT FILE FORMAT
The output model file describes a Markov model estimating the probability of a packet occurring given the last two packets.

The count of each continuation after a pair of successive packets is stored, and the ratios of these counts is used to calculate probabilities for the next packet.

The model is stored in JSON format, and also contains information about the packet rate and DNS traffic rate.

### Example ngram listing
The following listing shows a contrived example of a single ngram entry.

```
          "ngrams": {

             "ldap:0\tdcerpc:11": {

                 "lsarpc:77": 1,

                 "ldap:2": 370,

                 "ldap:3": 62,

                 "wait:3": 2,

                 "-": 1

             }, [...]

          }

```

This counts the observed continuations after an ldap packet with opcode 0 (a bind) followed by a dcerpc packet with opcode 11 (also a bind). The most common next packet is "ldap:2" which is an unbind, so this is the most likely packet type to be selected in replay. At the other extreme, lsarpc opcode 77 (lookup names) has been seen only once, and it is unlikely but possible that this will be selected in replay.

There are two special packet types here. "wait:3" refers to a temporary pause in the conversation, where the "3" pseudo-opcode indicates the length of the wait on an exponential scale. That is, a "wait:4" pause would be about 2.7 times longer that a "wait:3", which in turn would be similarly longer than a "wait:2".

The other special packet is "-", which represents the limit of the conversation. In the example, this indicates that one observed conversation ended after this particular ngram. This special opcode is also used at the beginning of conversations, which are indicated by the ngram "-\t-".

## VERSION
This man page is complete for version 4.22.10-alt1 of the Samba suite.

## SEE ALSO
traffic_replay(7).

## AUTHOR
The original Samba software and related utilities were created by Andrew Tridgell. Samba is now developed by the Samba Team as an Open Source project similar to the way the Linux kernel is developed.

The traffic_learner tool was developed by the Samba team at Catalyst IT Ltd.

The traffic_learner manpage was written by Tim Beale.


title: traffic_replay(7)

summary: Samba traffic generation tool.

text:
# traffic_replay(7)

## NAME
traffic_replay - Samba traffic generation tool.

## SYNOPSIS
traffic_replay [-F, --fixed-password <test-password>] [-T, --packets-per-second <number>] [-S, --scale-traffic <scale by factor>] [-r, --replay-rate <scale by factor>] [-D, --duration <seconds>] [--traffic-summary <output file>] [-I, --instance-id <id>] [-K, --prefer-kerberos] [-B, --badpassword-frequency <frequency>] [--dns-rate <rate>] [-t, --timing-data <file>] [--random-seed <seed>] [-U, --username user] [--password <password>] [-W --workgroup <workgroup>] [--realm <realm>] [-s, --config-file <file>] [-k, --kerberos <kerberos>] [--ipaddress <address>] [-P, --machine-pass] [--option <option>] [-d, --debuglevel <debug level>] [--conversation-persistence <0-1>] [--latency-timeout <seconds>] [--stop-on-any-error] {summary-file} {dns-hostname}

traffic_replay [-G, --generate-users-only] [-F, --fixed-password <test-password>] [-n, --number-of-users <total users>] [--number-of-groups <total groups>] [--average-groups-per-user <average number>] [--group-memberships <total memberships>] [--max-members <group size>] {dns-hostname}

traffic_replay {-c|--clean-up} {dns-hostname}

traffic_replay [-h, --help] [-V, --version]

## DESCRIPTION
This tool is part of the samba(7) suite.

This tool generates traffic in order to measure the performance of a Samba DC, and to test how well Samba will scale as a network increases in size. It can simulate multiple different hosts making multiple different types of requests to a DC.

This tool is intended to run against a dedicated test DC (rather than a live DC that is handling real network traffic).

Note that a side-effect of running this tool is that user accounts will be created on the DC, in order to test various Samba operations. As creating accounts can be very time-consuming, these users will remain on the DC by default. To remove these accounts, use the --clean-up option.

## OPTIONS
-h|--help

Print a summary of command line options.

summary-file

File containing the network traffic to replay. This should be a traffic-model (generated by traffic_learner). Based on this file, this tool will generate 'conversations' which represent Samba activity between a network host and the DC.

dns-hostname

The full DNS hostname of the DC that's being tested. The Samba activity in the summary-file will be replicated and directed at this DC. It's recommended that you use a dedicated DC for testing and don't try to run this tool against a DC that's processing live network traffic.

-F|--fixed-password <test-password>

Test users are created when this tool is run, so that actual Samba activity, such as authorizing users, can be mimicked. This option specifies the password that will be used for any test users that are created. Note that any users created by this tool will remain on the DC until you run the --clean-up option. Therefore, the fixed-password option needs to be the same each time the tool is run, otherwise the test users won't authenticate correctly.

random-seed

A number to seed the random number generator with. When traffic is generated from a model-file, use this option to keep the traffic consistent across multiple test runs. This allows you to compare the performance of Samba between different releases.

Traffic Model Options

When the summary-file is a traffic-model (produced by traffic_learner), use these options to alter the traffic that gets generated. -D|--duration <seconds>

Specifies the approximate duration in seconds to generate traffic for. The default is 60 seconds.

-T|--packets-per-second <number>

Generate this many packets per second, regardless of the traffic rate of the sample on which the model was based. This cannot be used with -S.

-S|--scale-traffic <factor>

Increases the number of conversations by this factor, relative to the original traffic sample on which the model was based. This option won't affect the rate at which packets get sent (which is still based on the traffic model), but it will mean more conversations get replayed. It cannot be combined with -T, which sets the traffic rate in a different way.

-r|--replay-rate <factor>

Replays the traffic faster by this factor. This option won't affect the number of packets sent, but it will squeeze them into fewer conversations, which may reduce resource usage.

--traffic-summary <output-file>

Instead of replaying a traffic-model, this option generates a traffic-summary file based on what traffic would be sent. Using a traffic-model allows you to scale the packet rate and number of packets sent. However, using a traffic-model introduces some randomness into the traffic generation. So running the same traffic_replay command multiple times using a model file may result in some differences in the actual traffic sent. However, running the same traffic_replay command multiple times with a traffic-summary file will always result in the same traffic being sent. For taking performance measurements over several test runs, it's recommended to use this option and replay the traffic from a traffic-summary file, or to use the --random-seed option.

--stop-on-any-error

Any client error causes the whole run to stop.

--conversation-persistence <0-1>

Conversation termination (as decided by the model) is re-interpreted as a long pause with this probability.

--latency-timeout <seconds>

Wait this long at the end of the run for outstanding reply packets. The number of conversations that have not finished at the end of the timeout is a failure metric.

--generate-users-only

Add extra user/groups on the DC to increase the DB size. By default, this tool automatically creates test users that map to the traffic conversations being generated. This option allows extra users to be created on top of this. Note that these extra users may not actually used for traffic generation - the traffic generation is still based on the number of conversations from the model/summary file. Generating a large number of users can take a long time, so it this option allows this to be done only once.

Note that the users created will remain on the DC until the tool is run with the --clean-up option. This means that it is best to only assign group memberships once, i.e. run --clean-up before assigning a different allocation of group memberships.

-n|--number-of-users <total-users>

Specifies the total number of test users to create (excluding any machine accounts required for the traffic). Note that these extra users simply populate the DC's DB - the actual user traffic generated is still based on the summary-file.

--number-of-groups <total-groups>

Creates the specified number of groups, for assigning the test users to. Note that users are not automatically assigned to groups - use either --average-groups-per-user or --group-memberships to do this.

--average-groups-per-user <average-groups>

Randomly assigns the test users to the test groups created. The group memberships are distributed so that the overall average groups that a user is member of matches this number. Some users will belong to more groups and some users will belong to fewer groups. This option is incompatible with the --group-membership option.

--group-memberships <total-memberships>

Randomly assigns the test users to the test groups created. The group memberships are distributed so that the total groups that a user is member of, across all users, matches this number. For example, with 100 users and 10 groups, --group-memberships=300 would assign a user to 3 groups on average. Some users will belong to more groups and some users will belong to fewer groups, but the total of all member linked attributes would be 300. This option is incompatible with the --average-groups-per-user option.

--max-members <group size>

Limit the largest group to this size, even if the other group options would have it otherwise.

--clean-up

Cleans up any users and groups that were created by previously running this tool. It is recommended you always clean up after running the tool.

-I|--instance-id <id>

Use this option to run multiple instances of the tool on the same DC at the same time. This adds a prefix to the test users generated to keep them separate on the DC.

-K|--prefer-kerberos

Use Kerberos to authenticate the test users.

-B|--badpassword-frequency <frequency>

Use this option to simulate users trying to authenticate with an incorrect password.

--dns-rate <rate>

Increase the rate at which DNS packets get sent.

-t|--timing-data <file>

This writes extra timing data to the file specified. This is mostly used for reporting options, such as generating graphs.

Samba Common Options

-d|--debuglevel=level

level is an integer from 0 to 10. The default value if this parameter is not specified is 1 for client applications. The higher this value, the more detail will be logged to the log files about the activities of the server. At level 0, only critical errors and serious warnings will be logged. Level 1 is a reasonable level for day-to-day running - it generates a small amount of information about operations carried out.

Levels above 1 will generate considerable amounts of log data, and should only be used when investigating a problem. Levels above 3 are designed for use only by developers and generate HUGE amounts of log data, most of which is extremely cryptic.

Note that specifying this parameter here will override the log level parameter in the /etc/samba/smb.conf file.

--debug-stdout

This will redirect debug output to STDOUT. By default all clients are logging to STDERR.

--configfile=<configuration file>

The file specified contains the configuration details required by the client. The information in this file can be general for client and server or only provide client specific like options such as client smb encrypt. See /etc/samba/smb.conf for more information. The default configuration file name is determined at compile time.

--option=<name>=<value>

Set the smb.conf(5) option "<name>" to value "<value>" from the command line. This overrides compiled-in defaults and options read from the configuration file. If a name or a value includes a space, wrap whole --option=name=value into quotes.

--realm=REALM

Set the realm name

-V|--version

Prints the program version number.

Credential Options

--simple-bind-dn=DN

DN to use for a simple bind

--password=PASSWORD

Password

-U USERNAME|--username=USERNAME

Username

-W WORKGROUP|--workgroup=WORKGROUP

Workgroup

--use-kerberos=desired|required|off

This parameter determines whether Samba client tools will try to authenticate using Kerberos. For Kerberos authentication you need to use dns names instead of IP addresses when connecting to a service. Note that specifying this parameter here will override the client use kerberos parameter in the /etc/samba/smb.conf file.

--ipaddress=IPADDRESS

IP address of the server

-P|--machine-pass

Use stored machine account password.

## OPERATIONS

### Generating a traffic-summary file
To use this tool, you need either a traffic-summary file or a traffic-model file. To generate either of these files, you will need a packet capture of actual Samba activity on your network.

Use Wireshark to take a packet capture on your network of the traffic you want to generate. For example, if you want to simulate lots of users logging on, then take a capture at 8:30am when users are logging in.

Next, you need to convert your packet capture into a traffic summary file, using traffic_summary.pl. Basically this removes any sensitive information from the capture and summarizes what type of packet was sent and when.

Refer to the traffic_summary.pl --help help for more details, but the basic command will look something like:

tshark -r capture.pcapng -T pdml | traffic_summary.pl > traffic-summary.txt

### Replaying a traffic-summary file
Once you have a traffic-summary file, you can use it to generate traffic. The traffic_replay tool gets passed the traffic-summary file, along with the full DNS hostname of the DC being tested. You also need to provide some user credentials, and possibly the Samba realm and workgroup (although the realm and workgroup may be determined automatically, for example from the /etc/smb.conf file, if one is present). E.g.

traffic_replay traffic-summary.txt my-dc.samdom.example.com -UAdmin%password -W samdom --realm=samdom.example.com --fixed-password=blahblah123!

This simply regenerates Samba activity seen in the traffic summary. The traffic is grouped into 'conversations' between a host and the DC. A user and machine account is created on the DC for each conversation, in order to allow logon and other operations to succeed. The script generates the same types of packets as those seen in the summary.

Creating users can be quite a time-consuming process, especially if a lot of conversations are being generated. To save time, the test users remain on the DC by default. You will need to run the --clean-up option to remove them, once you have finished generating traffic. Because the same test users are used across multiple runs of the tool, a consistent password for these users needs to be used - this is specified by the --fixed-password option.

The benefit of this tool over simply using tcprelay is that the traffic generated is independent of any specific network. No setup is needed beforehand on the test DC. The traffic no longer contains sensitive details, so the traffic summary could be potentially shared with other Samba developers.

However, replaying a traffic-summary directly is somewhat limited in what you can actually do. A more flexible approach is to generate the traffic using a model file.

### Generating a traffic-model file
To create a traffic-model file, simply pass the traffic-summary file to the traffic_learner script. E.g.

traffic_learner traffic-summary.txt -o traffic-model.txt

This generates a model of the Samba activity in your network. This model-file can now be used to generate traffic.

### Replaying the traffic-model file
Packet generation using a traffic-model file uses the same command as a traffic-summary file, e.g.

traffic_replay traffic-model.txt my-dc.samdom.example.com -UAdmin%password

By default, this will generate 60 seconds worth of traffic based on the model. You can specify longer using the --duration parameter.

The traffic generated is an approximation of what was seen in the network capture. The traffic generation involves some randomness, so running the same command multiple times may result in slightly different traffic being generated (although you can avoid this, by specifying the --random-seed option).

As well as changing how long the model runs for, you can also change how many conversations get generated and how fast the traffic gets replayed. To roughly double the number of conversations that get replayed, use --scale-traffic=2 or to approximately halve the number use --scale-traffic=0.5. To approximately double how quickly the conversations get replayed, use --replay-rate=2, or to halve this use --replay-rate=0.5

For example, to generate approximately 10 times the amount of traffic seen over a two-minute period (based on the network capture), use:

traffic_replay traffic-model.txt my-dc.samdom.example.com -UAdmin%password --fixed-password=blahblah123! --scale-traffic=10 --duration=120

### Scaling the number of users
The performance of a Samba DC running a small subset of test users will be different to a fully-populated Samba DC running in a network. As the number of users increases, the size of the DB increases, and a very large DB will perform worse than a smaller DB.

To increase the size of the Samba DB, this tool can also create extra users and groups. These extra users are basically 'filler' for the DB. They won't actually be used to generate traffic, but they may slow down authentication of the test users.

For example, to populate the DB with an extra 5000 users (note this will take a while), use the command:

traffic_replay my-dc.samdom.example.com -UAdmin%password --generate-users-only --fixed-password=blahblah123! --number-of-users=5000

You can also create groups and assign users to groups. The users can be randomly assigned to groups - this includes any extra users created as well as the users that map to conversations. Use either --average-groups-per-user or --group-memberships to specify how many group memberships should be assigned to the test users.

For example, to assign the users in the replayed conversations into 10 groups on average, use a command like:

traffic_replay traffic-model.txt my-dc.samdom.example.com -UAdmin%password --fixed-password=blahblah123! --generate-users-only --number-of-groups=25 --average-groups-per-user=10

The users created by the test will have names like STGU-0-xyz. The groups generated have names like STGG-0-xyz.

## VERSION
This man page is complete for version 4.22.10-alt1 of the Samba suite.

## SEE ALSO
traffic_learner(7).

## AUTHOR
The original Samba software and related utilities were created by Andrew Tridgell. Samba is now developed by the Samba Team as an Open Source project similar to the way the Linux kernel is developed.

The traffic_replay tool was developed by the Samba team at Catalyst IT Ltd.

The traffic_replay manpage was written by Tim Beale.