Commit Graph

112 Commits

Author SHA1 Message Date
Pekka Enberg
81aa2a8279 Merge "JMX proxy to support configuration file" from Amnon
"Scylla's configuration can change the listening address and port of the API,
the jmx proxy need to use this same configuration.

This series adds the ability to add a path to a yaml configuration file and the
jmx proxy would read its configuration from there.  Configuration from system
properties/command line is still supported and the configuration hirarchy is as
follow from highest to lowest:
* command line
* configuration file according to the hirarchy:
  - command line
  - SCYLLA_CONF
  - SCYLLA_HOME
  - relative conf directory
* default values

The configuration definition was moved to a configuraion class that responsible
for getting the information from the command line and the configuration file."
2015-11-11 09:08:08 +02:00
Amnon Heiman
27c0eb8c99 StorageService: move should pass the parameter to the API
When calling the API move method, the proxy should pass the new_token
parameter.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Signed-off-by: Pekka Enberg <penberg@scylladb.com>
2015-11-11 08:53:27 +02:00
Amnon Heiman
43e07fda0e Use the configuration object
This patch changes the APIClient to read the connection string from the
configuration object.

Main uses the same configuraion API to print it's connecting message and
call the configuration setup.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-08 12:12:27 +02:00
Amnon Heiman
39b50f63c7 Adding a configuration object for the jmx proxy
This patch adds a configuration object to the jmx proxy that support
both the system/command line based properties and it accept a yaml
configuration file. The later options allows the jmx to read scylla
configuration file and connect to it based on this configuration.

The configuration file reader uses a yaml parser that was added to the
pom.xml

If no configuration file is found in the command line, it would look for
SCYLLA_CONF then SCYLLA_HOME then for relative 'conf' directory

Signed-off-by: Amnon Heiman <amnon@scylladb.com>

need merge apiconfig
2015-11-08 12:12:27 +02:00
Avi Kivity
72f6f5dab4 Merge "Enabling nodetool netstats" from Amnon
"This series adds the jmx implementation to enable netstats.
After this series netstats should complete successfuly.
A run example:

$ ./bin/nodetool netstats
Mode: NORMAL
repair 397c91a0-8205-11e5-83e4-000000000001
repair 3977d5ba-8205-11e5-83e4-000000000001
repair 3977d624-8205-11e5-83e4-000000000001
repair 397c8fc8-8205-11e5-83e4-000000000001
.......
......
repair 3977d502-8205-11e5-83e4-000000000001
Read Repair Statistics:
Attempted: 1
Mismatch (Blocking): 0
Mismatch (Background): 0
Pool Name                    Active   Pending      Completed
Commands                        n/a         0          21182
Responses                       n/a         0            597"
2015-11-08 11:48:40 +02:00
Amnon Heiman
9b14550d0a StorageService: register the StreamManagerMBean
This patch adds the registration of StreamManagerMBean to
StorageService, similiar to the way it is done in origin.

After this patch the StreamManager will be available via Jconsole.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 11:33:40 +02:00
Amnon Heiman
3cb168bed3 MessagingService: Add tasks statistics
This patch adds the implementation of:
getResponsePendingTasks()
getResponseCompletedTasks()
getDroppedMessages()

The implementation is based on the messaging_service API that defined in
messaging_service.json.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 11:33:40 +02:00
Amnon Heiman
20778b2df6 Add StreamManagerMBean with its subclasses
The StreamManager getStreams returns an hirarchy of classes. This patch
import StreamManagerMBean with the class hirarchy and add an
implementation to StreamManager.

The implementation is based on the stream_manager API.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 11:28:44 +02:00
Amnon Heiman
1db077618d APIClient: Add getMapStringIntegerValue and getMapStringLongValue
This patch adds two map reader function to the APIClient, one that parse
map<String,Integer> and one for map<String,Long>

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-11-03 11:18:33 +02:00
Amnon Heiman
00694ce40a StorageService: getLoadString should return units
This address issue #512

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-10-29 15:54:29 +02:00
Amnon Heiman
649922fed8 Updating the http client and support API exception
The API returns errors with an HTTP code 400, 404 or 500 depending on
the cause with a json object that contains the failure reason.  The
error message should be populate to the JMX calling client, translated
to the appropriate exception.

This patch adds the ability to detect API failure and throw a runtime
exception with the returned message.

It is up to the calling method what to do with the exception, if it
would do nothing, the calling client would get a RuntimeException,
depends on origin MBean definition, the caller can catch the exception
and throw a specific kind.

Note that any exception that is thrown must be known to the JMX client
or it will not be able to process it.

As a first step, this patch replaces the jersey client to the newer
version under glassfish, which has an easy way of getting a Reply
object, and check its status before returning the results.

The only difference in the method that uses the APIClient is the use of
MultivaluedHashMap.

The following MBean implementation where changed

ColumnFamily
CompactionManager
Gossiper
EndpointSnitchInfo
CacheService
StorageProxy
StorageService

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2015-10-28 11:25:37 +02:00
Amnon Heiman
a707b487f0 StorageService: remove node should use hostid and not token
There is a confusion in origin, the MBean declare token as the parameter
to remove, but the implementation actually uses host id.

This patch modify scylla implementation to pass a host id as the
parameter to remove.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-26 10:31:43 +02:00
Pekka Enberg
dc76d2550e Merge "Support deprecated method in StorageProxy" from Amnon
"This series adds latency related depricated methods to the storage
 proxy.

 The implmenetation mimic origin, in which the depricated methods calls
 the counters that replaces them."
2015-10-26 09:15:20 +02:00
Amnon Heiman
ae26226339 ClientRequestMetrics: pass the url similiar to LatencyMetrics
In LatencyMetrics the URL is passed without the ending slash, this
patch use the same notation in ClientRequestMetrics.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-25 13:36:10 +02:00
Avi Kivity
ee7f5354b1 Avoid non-final variables in anonymous classes
Required by Java 7.
2015-10-25 13:22:05 +02:00
Amnon Heiman
392765fa2e StorageProxy: Add depricated method implementation based on metrics
This follow origin by adding the implementation for the depricated
metrics methods.

Similiar to origin, the implementation calls the implementation in the
metrics.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-22 10:43:29 +03:00
Pekka Enberg
b11c108fcf Remove obsolete comments from ColumnFamilyStore 2015-10-21 14:28:17 +03:00
Pekka Enberg
8e0fb98fc4 Merge "Adding estimated histogram support for nodetool cfhistogram" from Amnon
"This series together with the cfhistogram series in scylla adds the
 missing functionality so that nodetoold cfhistogram would work.

 After both series will be apply an execution example is:

 ./bin/nodetool cfhistograms keyspace1 standard1
 keyspace1/standard1 histograms
 Percentile  SSTables     Write Latency      Read Latency    Partition Size        Cell Count
                               (micros)          (micros)           (bytes)
 50%             0.00           6866.00        4866323.00               310                 5
 75%             0.00           8239.00       10090808.00               310                 5
 95%             0.00          20501.00       17436917.00               310                 5
 98%             0.00          35425.00       25109160.00               310                 5
 99%             0.00          51012.00       25109160.00               310                 5
 Min             0.00           2300.00         654950.00               259                 5
 Max             0.00       20924300.00       25109160.00               310                 5"
2015-10-21 14:26:40 +03:00
Amnon Heiman
3667682075 ColumnFamilyStore: add support for estimated latency
This patch uses the estimated latency that was added to the column
family metrics to get the recent and estimated latency.

It follows the same logic as origin does to call the logic in metrics.

The following method implementation will be added:

getMemtableColumnsCount
getRecentSSTablesPerReadHistogram
getSSTablesPerReadHistogram
getLifetimeReadLatencyHistogramMicros
getRecentReadLatencyHistogramMicros
getRecentReadLatencyMicros
getLifetimeWriteLatencyHistogramMicros
getRecentWriteLatencyHistogramMicros
getRecentWriteLatencyMicros
getRangeCount
getTotalRangeLatencyMicros
getLifetimeRangeLatencyHistogramMicros
getRecentRangeLatencyHistogramMicros
getRecentRangeLatencyMicros

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:22:06 +03:00
Amnon Heiman
db3b3cdeee ColumnFamilyMetrics: add estimted per read histogram support
This uses the recent estimated histogram and the API based estimated
histogram to support the sstable per read recent and total estimated
histogram.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:22:06 +03:00
Amnon Heiman
960aa6f509 LatencyMetrics: Support totalLatencyHistogram and recentLatencyHistogram
This adds the depricated total and recent estimated histogram.
It uses the new RecentEstimatedHistogram for the recent value and the
API based estimated histogram for the total latency.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:22:06 +03:00
Amnon Heiman
6df716e5f7 Add an EstimateHistogramWrapper
The EstimatedHistogramWrapper is a helper class that holds the API
related data, so that a class that uses an EstimatedHistogram can
replace it with the wrapper and keep most of its code as is.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:21:53 +03:00
Amnon Heiman
a7c30493cd APIClient to support EstimatedHistogram
This patch modify the CacheEntry to support both String and
EstimatedHistogram.

It is possible to add more supported types in the future when needed.

In the APIClient, the cache will now support both String and
EstimatedHistogram in a similiar way.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>

apiclient need to merge to cache
2015-10-21 14:20:48 +03:00
Amnon Heiman
931571c3ed EstimatedHistogram: Add constructor from data
This patch allows to create an EstimatedHistogram from an array of data
value.

It will be used by the APIClient to return EstimatedHistogram

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:20:04 +03:00
Amnon Heiman
a568e52cb5 Adding the RecentEstimatedHistogram
Some of the jmx methods uses the notion of recent estimated histogram.
In origin the implementation uses an estimated histogram and clean the
histogram values on each call.

The RecentEstimatedHistogram mimic this behaviour, it store the latest
values of the last call. In each call new values are stored in the
histogram and the results is the delta between the last two calls.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 14:19:57 +03:00
Amnon Heiman
3bf092c83b StorageService: handle loadNewSStable and forceKeyspaceCleanup
keyspace cleanup was changed from GET to POST and load sstable was
missing the keyspace name.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-21 12:27:59 +03:00
Nadav Har'El
f05c6709dd Fix "nodetool repair" error
The existing scylla-jmx code had reversed logic for the success of the repair:
it reported to "nodetool repair" a failure when the repair was successful :-)

Note that "nodetool repair" waits until a FINISHED notification, and then reports
a failure if it previously got any SESSION_FAILED notification; So if repair was
successful, all we need to do is to avoid sending a "SESSION_FAILED" message.
But we don't need to send any additional "SESSION_SUCCESS" message to signal
success. That message type is only used to report progress to the user (a
"session" is part of the repair work, so seeing sessions completing shows
progress), but because Scylla doesn't support this progress report yet, we
can't send these notifications yet, and there's no point in sending one such
message at the end - it's only confusing (especially when the text is the same
as that of the FINISHED message).

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
2015-10-15 09:50:35 +03:00
Amnon Heiman
6ed255f2e3 EndpointSnitchInfo: Cache results for data center and rack
This cache the get data center and get rack results for 10s, it has a
direct impact on nodetool status and nodetool ring

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-07 10:24:50 +03:00
Amnon Heiman
929d0aedd4 EndpointSnitchInfo formatting 2015-10-07 10:14:49 +03:00
Amnon Heiman
c5c0fb41fe APIClient: Adding cache support to the APIClient
Some operations are not changing frequently and are called multiple time
during a nodetool execution.

This patch adds the ability to cache results for a define period of time
(typically it will be for a few seconds) so that during the same
nodetool command call, the results will be retrieved from the cache.

It is currently only implemented for string values, other commands will
be added when needed.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-07 10:13:05 +03:00
Amnon Heiman
38af24d49d Adding CacheEntry to be used by the APIClient cache 2015-10-07 09:43:37 +03:00
Pekka Enberg
a90e080ee6 Merge "Complete the nodetool cfstats support" from Amnon
"This series complete the scylla series to support the nodetool cfstatus support.

After this series it will be possible to call nodetoold cfstatus and get a meaningfull output.

An output example:
./bin/nodetool cfstats keyspace1
Keyspace: keyspace1
        Read Count: 87657
        Read Latency: 1.1418900715287998 ms.
        Write Count: 87177
        Write Latency: 0.022303761313190406 ms.
        Pending Flushes: 0
                Table: standard1
                SSTable count: 8
                SSTables in each level: [               Space used (live): 92356832
                Space used (total): 92356832
                Space used by snapshots (total): 0
                Off heap memory used (total): 106430512
                SSTable Compression Ratio: 0.0
                Number of keys (estimate): 328672
                Memtable cell count: 100000
                Memtable data size: 84800254
                Memtable off heap memory used: 105906176
                Memtable switch count: 4
                Local read count: 92854
                Local read latency: 1.039 ms
                Local write count: 93880
                Local write latency: 1.045 ms
                Pending flushes: 0
                Bloom filter false positives: 0
                Bloom filter false ratio: 0.00000
                Bloom filter space used: 208416
                Bloom filter off heap memory used: 524336
                Index summary off heap memory used: 0
                Compression metadata off heap memory used: 0
                Compacted partition minimum bytes: 259
                Compacted partition maximum bytes: 310"
2015-10-06 12:23:09 +03:00
Amnon Heiman
a8292e3e09 ColumnFamilyMetrics: add estimated row count histogram
This adds the estimated row count histogram support to
ColumnFamilyMetrics

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-04 12:54:31 +03:00
Amnon Heiman
54fe3d7ccf ColumnFamilyMetrics: Use schedule for the column family timer
The logic of that timer, should be that after some defined time from the
previous request a new one will be sent, the actuall rate is
meaningless and only cause delyed request to be sent in a higher
frequency after the delayed reponse was returned.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-04 12:49:51 +03:00
Amnon Heiman
f16c49a860 LatencyMetrics: combine the different constructors
This combine the different constructor into one constructor with the
logic and another one that calls it with a default value.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-04 12:48:26 +03:00
Amnon Heiman
9c7ab3fcf6 APIHistogram: Use the new histogram parameter name
The use of mean and variance as histogram parameter names makes more
sense.

This also make it safe to use an empty histogram that holds no samples.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-10-04 12:46:44 +03:00
Amnon Heiman
d67e6e60fc EndPointSnitchInfo: Support null as host name
In origin an undocumented feature (bug?) is that passing null as a host
name for getRack and getDatacenter returns the rack or datacenter
according to the loopbck address.

This follow the same behaviour, so when the host is null, the function
will not fail but will call the API with the local loopback address

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-09-29 09:30:08 +03:00
Amnon Heiman
5feb4d3841 Adding depricated implementation to column_family
Some of the depricated method in column family are replaced by counters
in Origin. Still, some tools, like nodetool uses the depricated API.

This adds an implementation to some of the depricated method that is
based on the column family counters similiar to origin.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-09-17 13:43:19 +03:00
Amnon Heiman
713f8a5c75 Shuting down the the proxy after MAX_RETRY
This uses the column family timer to check that the API is still
available.

After MAX_RETRY (currently set to 30s) the proxy will shutdown.
This address issue #2

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-09-17 13:38:01 +03:00
Amnon Heiman
3e326bf2d0 APIClient: Fixing a problem with getString
The javax does not handle parsing JsonValue directly. So it is replaced
for the getString with a direct removal of the quotations.

The alternative is to replace the javax parser with a different one.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
2015-09-14 13:32:15 +03:00
Amnon Heiman
cce767e47e APIClient: getString to return parsed json value
getString should return the string value without the sarounding quotes.

For getRaw was introduce that return the API values in its raw value,
and getString was changed to use the json parser to get the string
values without the quotes.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudus-systems.com>
2015-09-11 09:25:37 +03:00
Amnon Heiman
fe3b786053 Urchin-JMX: Adding the EndpointSnitchInfo implementation
This adds the call to the API for the EndpointSnitch, the url are based
on the endpoint_snitch_info.json swagger definition file.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudus-systems.com>
2015-09-11 09:25:00 +03:00
Pekka Enberg
db6d48a695 Merge "Adding notification for the repair command" from Amnon
"The API is currently not supporting notification, but the repair
 command that perform via the nodetool relays on the notification to know
 when the command terminate.

 This series adds support for the repair notification, based on a timer
 and periodically check if a current repair command was terminated."
2015-08-31 09:41:01 +03:00
Amnon Heiman
67dca4da9d StorageService: Add notification suport for the repair command
This patch adds a notification support for the repair in StorageService.

When a repair command starts a timer is set to check the status of the
repair, when the repair complets it sends notification for the
successful or fail of the repair.

Because jconsole doesn't run method with variable number of parameters
an additional MBean method was added for async repair with only a
keyspace as its parameter.

All the forceAsycRepair methods are mapped to the asyncRepair metod that
replaces them in 2.2, when options will be supported in the repair, it
will be added to the jmx.

After this patch it is possible to register in the jconsole for
notification, perform a forceAsyncRepair and get the notification that
the repair complets
2015-08-27 21:02:28 +03:00
Amnon Heiman
e96e62b6bd Add the RepairParallelism enum
This takes the RepairParallelism enum from origin

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-27 20:46:02 +03:00
Amnon Heiman
6fc2c5c720 APIClient add postInt
Sometimes a post command need to return, this adds a method to perform a
post command that returns an int value.

The general postGetVal, can be used for other types if needed.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-27 20:46:02 +03:00
Amnon Heiman
2cfe02acb8 LatencyMetrics: Support path parameter
This change adds path parameter support, for the cases that the path
parameter is the last on the path, for example in the ColumnFamily case

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-24 17:13:34 +03:00
Amnon Heiman
0ae282b548 Add the getEstimatedRowSizeHistogram implementation
The getEstimatedRowSizeHistogram is based on the column_family.json that
defined in the API.

The implementation in the MBean and in the metrics is now works with the
get histogram as long array method in the client.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-24 17:13:34 +03:00
Amnon Heiman
bde40aed53 APIClient: Add a getter for estimated histogram buckets
This method returns the estimated histogram buckets. It is based on the
module definied in utils.json in the API.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-24 17:13:34 +03:00
Amnon Heiman
b53be3a4ec StorageService: Add the effectiveOwnership and getOwnership implementation
This adds the implementation of the effectiveOwnership and getOwnership
in StorageService. It uses the API that defined in storage_service.json.

After this patch the effectiveOwnership and getOwnership will be
available via jconsole.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
2015-08-24 13:12:33 +03:00