Commit Graph

555 Commits

Author SHA1 Message Date
Amnon Heiman
71170f5713 CompactionMetrics: use the pending compaction API (#75)
The PendingTasksByTableName metric should use the pending_tasks_by_table
API to get the real value of the pending compaction.

Fixes #74

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2019-08-05 14:12:48 +03:00
Amnon Heiman
ff0723abc6 ColumnFamilyStore: Mbean API support the hex format param (#69)
Cassandra 3.0 version of the JMX added a parameter that allows accepting
the parameter as hex.

This breaks the current implementation with a NoSuchMethodException.

This patch adds the missing implementation.

For a full support, a follow up patch in Scylla is needed, but for the
current functionality it would work.

After this patch usage example:

nodetool getsstables keyspace1 standard1 39303138374b4d343830

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2019-07-29 10:09:04 +03:00
Calle Wilund
cb42205061 service: Add objects for deserializing sstable_info json
Objects + serial logic to automate the transform of
scylla REST json object for sstable_info into
compositedata that can be consumed by nodetool
2019-07-24 14:31:10 +00:00
Calle Wilund
b2f3eeee05 scylla-apiclient: Add Date json serializer helper
To handle RFC8601 formattedd dates in JAXB
2019-07-24 14:30:02 +00:00
Calle Wilund
d8efa60ab7 APIClient: Add jackson JSON serializer support to client object
Allows java ws to deserialize json objects directly.
2019-07-24 14:28:38 +00:00
Calle Wilund
bbc817013e apiclient/pom.xml: Add jackson JSON support libs for REST client 2019-07-24 14:27:56 +00:00
Calle Wilund
263735379e scylla-jmx: Remove depdendency-reduced-pom.xml from tracking
This file is (re-)generated by maven on occasions. It should
not be version controlled.

Add to .gitignore as well.
2019-07-23 11:05:34 +03:00
Amnon Heiman
f0d2df3d15 StorageProxy.java: Add view write metrics
nodetool proxyhistograms command look for the view write metric.

While we do not report that metric yet, we still want the command to
succeed.

After this patch:
$ nodetool proxyhistograms
proxy histograms
Percentile       Read Latency      Write Latency      Range Latency   CAS Read Latency  CAS Write Latency View Write Latency
                     (micros)           (micros)           (micros)           (micros)           (micros)           (micros)
50%                    326.00             110.00             424.50               0.00               0.00               0.00
75%                   1253.00             193.25             877.75               0.00               0.00               0.00
95%                   2935.90            1007.25            5182.55               0.00               0.00               0.00
98%                   3100.00            1040.60            5492.00               0.00               0.00               0.00
99%                   3100.00            1058.00            5492.00               0.00               0.00               0.00
Min                     34.00               9.00              36.00               0.00               0.00               0.00
Max                   3100.00            1058.00            5492.00               0.00               0.00               0.00

See scylladb/scylla#4470

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20190514063316.28040-1-amnon@scylladb.com>
2019-07-21 19:20:37 +03:00
Amnon Heiman
c7bce65919 APIMBeanServer: Support both Table and Tables as metric name
Fixes #71

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2019-07-17 10:56:44 +03:00
Pekka Enberg
4303f06426 Merge "Make API client a separate module for reuse" from Ľuboš
There's an effort to implement a version of "nodetool" that uses
Scylla's REST API directly, Let's make the API client a separate module,
so nodetool can use it.

* 'scylla-apiclient' of https://github.com/tarzanek/scylla-jmx:
  fix README for building instructions
  trigger build from parent maven to have the local repo properly set up
  cleanup commented implicit steps in mvn
  make scylla-apiclient a separate module so the jar can be reused
2019-07-10 22:31:12 +03:00
Lubos Kosco
183eb6158a fix README for building instructions 2019-07-08 11:02:45 +02:00
Lubos Kosco
4296c7d3ae trigger build from parent maven to have the local repo properly set up 2019-07-08 10:54:04 +02:00
Lubos Kosco
222990d821 cleanup commented implicit steps in mvn 2019-07-08 10:04:24 +02:00
Lubos Kosco
91ae4ec8ee make scylla-apiclient a separate module so the jar can be reused 2019-07-01 17:33:08 +02:00
Amnon Heiman
9dae28e2f0 ColumnFamilyStore: finishLocalSampling should respect count limit
When calling nodetool toppartitions with size limit, finishLocalSampling
should respect that and limit the number of the results.

Example:
$ nodetool toppartitions -k 2 keyspace1 standard1 20
WRITES Sampler:
  Cardinality: ~2 (256 capacity)
  Top 2 partitions:
	Partition                Count       +/-
	38333032394d4f4d5030         4         3
	4e353937383137503330         4         3

READS Sampler:
  Cardinality: ~2 (256 capacity)
  Top 2 partitions:
	Nothing recorded during sampling period...

Fixes #66

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2019-06-23 15:31:18 +03:00
Amnon Heiman
2fac82434b APIClient: delete command should check for errors
delete commands do not return a value, still, it is possible that the
command will return a value different than OK.

In such a case, the error should be propagate to the caller via an
exception.

Fixes #65

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20190618135312.2776-1-amnon@scylladb.com>
2019-06-18 18:56:30 +03:00
Takuya ASADA
eda6f4e1a8 dist/debian: run 'systemctl daemon-reload' automatically on package install/uninstall
Since we cannot use dh --with=systemd because we don't want to
automatically enabling systemd units, manage them by our setup scripts,
we have to do 'systemctl daemon-reload' manually.
(On dh --with=systemd, systemd helper automatically provides such
scirpts)

See scylladb/scylla-enterprise#825

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190618000414.29142-1-syuu@scylladb.com>
2019-06-18 15:46:29 +03:00
Takuya ASADA
f73da49f62 dist: merge /usr/lib/scylla to /opt/scylladb
Since scylla-jmx uses /usr/lib/scylla/jmx for program directory, we also
need to move them under /opt/scylladb.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190618122451.27721-1-syuu@scylladb.com>
2019-06-18 15:43:53 +03:00
Calle Wilund
512638ed6e APIMBeanServer: Handle nodeprobe wildcard queries in CF refresh
Fixes #63
Message-Id: <20190311082942.3310-2-calle@scylladb.com>
2019-05-05 18:10:37 +03:00
Calle Wilund
5f974bc2bb ColumnFamilyStore: Propapgate exception cause in sampling wait
Message-Id: <20190311082942.3310-1-calle@scylladb.com>
2019-05-05 18:10:37 +03:00
Takuya ASADA
5e50090bfd dist: merge product name parameter on single place
When we add product name customization, we mistakenly defined the
parameter on each package build script.
Number of script is increasing since we recently added relocatable
python3 package, we should merge it in single place.

Also we should save the parameter on relocatable package, just like
version-release parameters.

So move the definition to SCYLLA-VERSION-GEN, save it to
build/SCYLLA-PRODUCT-FILE then archive it to relocatable package.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190422105304.23454-1-syuu@scylladb.com>
2019-04-22 13:55:53 +03:00
Piotr Jastrzebski
cb1ac4a58c Capture heap dump on OutOfMemoryError
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <ae90bb6e4b18f51376d2e7009078bf8c8e6ed7fd.1552407088.git.piotr@scylladb.com>
2019-03-26 16:24:35 +02:00
Calle Wilund
da21305989 StorageService: Include the arguments in "upgrade" call.
Message-Id: <20190219133431.29009-1-calle@scylladb.com>
2019-02-27 10:33:43 +02:00
Amnon Heiman
27313ee2c4 ColumnFamilyStore: Add an implementation for table sampling
This patch adds the implementation for begin and finish local sampling
of a column family.

There is a difference in the implementation of Cassandra API and Scylla.

In Cassandra and the JMX an external source start and stop the sampling.

In Scylla, a single API call start the sampling and return with the
result. In Scylla the API call always return sampling of the read and of
the writes.

To bridge the difference, the begin sampling command will use a Future
when calling the API. The finish method will wait for the future to end.

Because of the different implementation, it is possible that two
consecutive calls will be made to start sampling one for the read and
one for the write, similarly, two calls will be made to finish for read
and write.

The implementation would ignore the second call to start and will
store the result, so the second call to finish will be served from the
stored result.

Note, that the use of future is only for safety, the way we expect it to
work, the caller to the begin sampling will sleep anyhow while waiting
for the result.

To avoid breaking the MBean compatibility we piggyback the duration on
top of the sampler string.

If no duration is given, a default duration will be taken, this is also
just as a precaution, we will modify the nodetool implementation to
pass that information.

There is a known issue with cardinality, that will need to be addressed.
Also we return a value in the raw column to match what Cassandra JMX
returns, but it's a duplication of the partition key.

See scylladb/scylla#2811

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20190128143505.5241-1-amnon@scylladb.com>
2019-02-03 12:40:04 +02:00
Takuya ASADA
d4493295ff dist/debian: skip running dh_strip_nondeterminism
On some Fedora environment dh build tries to run
dh_strip_nondeterminism, and fails sice Fedora does not provide such
command.

To prevent the build error we need to skip it.

Fixes #62
See 5bf9a03d65

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20190125223351.20381-1-syuu@scylladb.com>
2019-01-28 09:12:50 +02:00
Calle Wilund
9eec9eabf6 scylla-jmx: Make scylla-jmx compatible with jdk9+
Adds explicit maven dependecies for libraries
removed from JDK.
Removes reflection calls forbidden in jdk9+.

Message-Id: <20181120142550.22852-1-calle@scylladb.com>
2018-11-21 13:00:24 +02:00
Takuya ASADA
854e6072a2 dist/redhat: prevent build error on older Fedora/CentOS
Current scylla.spec fails build on Fedora 27, since python2-pystache is
new package name that renamed on Fedora 28.
But Fedora 28's python2-pystache has tag "Provides: pystache",
so we can depends on old package name, this way we can build scylla.spec both
on Fedora 27/28.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181028175757.32224-1-syuu@scylladb.com>
2018-10-29 11:37:25 +02:00
Takuya ASADA
21e22d4e1a dist/redhat: minor fixes for relocatable .rpm
- we don't use mock anymore, so drop mock directory
- build_rpm.sh usage need to update
- build_rpm.sh should install rpmbuild

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181024222136.3332-2-syuu@scylladb.com>
2018-10-25 10:29:30 +03:00
Takuya ASADA
9ed8a01519 dist/debian: minor fixes for relocatable .deb
- we don't use pbuilder anymore, so drop pbuilderrc
- on build_deb.sh is_debian / is_ubuntu functions are unused now, drop them

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181024222136.3332-1-syuu@scylladb.com>
2018-10-25 10:29:27 +03:00
Avi Kivity
df2dee2402 Merge "relocatable package support for jmx" from Takuya
"
This patchset adds relocatable package support for scylla-jmx, and also
support generating .rpm/.deb from relocatable package.

 - Scripts are based on relocatable .rpm/.deb support patchset for main repo
   (not merged https://github.com/syuu1228/scylla/tree/reloc_rpmdeb_v4)
 - Single .rpm package provided for CentOS7/Fedora(unofficial)
 - Single .deb package provided for Ubuntu 14/16/18, Debian 8/9
"

* 'reloc_v1' of https://github.com/syuu1228/scylla-jmx:
  dist/debian: use relocatable package to produce .deb
  dist/redhat: use relocatable package to produce .rpm
  reloc: add support relocatable package
2018-10-24 11:51:34 +03:00
Takuya ASADA
e9bfbedfed dist/debian: use relocatable package to produce .deb 2018-10-24 11:45:11 +09:00
Takuya ASADA
5edfedf642 dist/redhat: use relocatable package to produce .rpm 2018-10-24 02:04:49 +00:00
Takuya ASADA
92847e3381 reloc: add support relocatable package
To align build system with scylla main repo, adding relocatable package
support.

On scylla-jmx, we don't provide libraries and linker since it's Java
program, just contains .jar file and dist/ directory.
2018-10-24 02:02:25 +00:00
Calle Wilund
ca3fa8de20 scylla-jmx: Fix tablemetricsobjectname breakage
Fixes #57

The usage of TableMetricsObjectName-yada-yada relies on translating the
"fake" objectname to a canonical one on remote
publication/serialization. However, the implementation of
ObjectName.getInstance has changed in JDK (micro) updates so it no
longer applies overridable methods -> wrong name published.

Fix by doing explicit ObjectName instansiation.
Message-Id: <20181023132005.23099-1-calle@scylladb.com>
2018-10-23 16:30:29 +03:00
Takuya ASADA
74fa1a40ca dist/debian: install GPG key for cross-building
We found on some Debian environment Ubuntu .deb build fails with
gpg error because lack of Ubuntu GPG key, so we need to install it before
start pbuilder.
Same as on Ubuntu, it needs to install Debian GPG key.

See scylladb/scylla#3823

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20181008110724.18335-1-syuu@scylladb.com>
2018-10-08 15:34:18 +03:00
Takuya ASADA
cd1c79f90f dist/debian: support package renaming on build script
To automatically rename packages on enterprise release, added package name
prefix as a variable on build_deb.sh.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180828010445.11920-2-syuu@scylladb.com>
2018-08-28 09:26:02 +03:00
Takuya ASADA
a27d9601f5 dist/redhat: support package renaming on build script
To automatically rename packages on enterprise release, added package name
prefix as a variable on build_rpm.sh.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180822072105.9420-2-syuu@scylladb.com>
2018-08-22 11:05:00 +03:00
Calle Wilund
c6aee9f63e scylla-jmx: Add "PendingTasksByTableName" gauge to CompactionMetrics
Required by origin 3.11 nodetool.

Message-Id: <20180801084545.23239-1-calle@scylladb.com>
2018-08-01 14:25:06 +03:00
Calle Wilund
9c3ac3e547 scylla-jmx: Update JMX interfaces to origin 3.11
Almost 100% null implementations, which is ok for most purposes
currently used by scylla. Some of these new calls (like dropped
mutations etc) should perhaps however be implemented.

Tested with the nodetool dtests. So sparsely.

Needed when/if scylla-tools-java is upgraded to origin 3.11,
otherwise noodtool breaks.

Message-Id: <20180730113741.14952-1-calle@scylladb.com>
2018-07-30 15:47:43 +03:00
Avi Kivity
b4d983b45a dist: redhat: fix up bad file ownership of rpms/srpms
mock outputs files owned by root. This causes attempts
by scripts that want to junk the working directory (typically
continuous integration) to fail on permission errors.

Fixup those permissions after the fact.
Message-Id: <20180719163258.4393-1-avi@scylladb.com>
2018-07-26 08:17:59 +03:00
Takuya ASADA
d6c408445e dist/debian/build_deb.sh: make build_deb.sh more simplified
Use is_debian()/is_ubuntu() to detect target distribution, also install
pystache by path since package name is different between Fedora and
CentOS.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180703183211.3455-1-syuu@scylladb.com>
2018-07-04 10:12:31 +03:00
Takuya ASADA
2af17c1f53 dist: simplified build script templates
Currently we are using *.in files for templates, applying parameters by sed
command one-by-one.
This patch will replace them by Mustache, it's simple and easy syntax template
language.

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180606183113.25275-1-syuu@scylladb.com>
2018-06-10 19:37:26 +03:00
Takuya ASADA
b8394f677b dist/debian: run update-ca-certificates to avoid security exception on Ubuntu 18.04
On Ubuntu 18.04 build fails by java.security.InvalidAlgorithmParameterException,
while downloading .pom file from HTTPS URL.
Looks like it's ca-certs problem, can fix by running update-ca-certificates.

Fixed #52

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20180601092954.28319-1-syuu@scylladb.com>
2018-06-03 10:51:11 +03:00
Piotr Jastrzebski
1ad2ba8507 TableRepository: wrap initial repository
Before we were discarding the initial repository while
overriding it with TableRepository. This was a mistake that
caused dtests to fail. Proper solution is to keep the initial
repository inside TableRepository. That way whatever was registered
at the time of JmxMBeanServer creation is still handled properly.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <22181859012fd20ddf37e049a145bc94a3a91a33.1527844328.git.piotr@scylladb.com>
2018-06-02 20:42:00 +03:00
Avi Kivity
71f857b1de Merge "Reduce memory usage and speed up start time" from Piotr
"
Use more memory efficient MBeans repository and remove quadratic behaviour on startup.

This reduces memory usage for 2000 tables from 127M to 82M and reduces start time
from 270 seconds to 2 seconds.

Changes since last version:
1. Fix registered map to handle multiple JMX servers and to properly deregister mbeans
2. Clean up TableRepository code.
"

* 'speedup_2' of https://github.com/haaawk/scylla-jmx:
  Use more efficient MBeans repository
  Remove unnecessary quadratic algorithm from MetricsMBean.register
2018-05-21 11:13:15 +03:00
Avi Kivity
e27312df10 Merge "Reduce memory usage" from Piotr
"
Introduce more memory efficient version of ObjectName and use it in JMX Server.
The original version stores the same data multiple times in different forms.
Big part of data is shared by multiple instances of ObjectName.
Original class keeps a separate copy for each instance.
The new version keeps only one copy that's shared by all instances.
"

* 'speedup_1' of https://github.com/haaawk/scylla-jmx:
  Introduce and use TableMetricObjectName
  Ensure regular ObjectName is returned to remote callers
  Use JmxMBeanServer instead of MBeanServer
2018-05-21 11:10:22 +03:00
Piotr Jastrzebski
862aea4a33 Use more efficient MBeans repository
Default implementation stores MBeans in the following map:

<domain name> -> (<properties as a single string> -> NamedObject)

This is problematic because NamedObject contains ObjectName that
has both domain and properties inside itself.

This means we're storing the same data twice.

For domain "" we want to store MBeans in a more compact way using map:

ObjectName -> DynamicMBean

which is equivalent to NamedObject.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-05-16 16:53:09 +02:00
Piotr Jastrzebski
5cba016962 Remove unnecessary quadratic algorithm from MetricsMBean.register
Before this change it was taking JMX Server 270 seconds to start
when Scylla had 2000 tables. After the change it takes only 2 seconds.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-05-16 16:21:21 +02:00
Piotr Jastrzebski
455f5717ea Introduce and use TableMetricObjectName
This is a new extention of ObjectName that uses less memory.

TableMetricNameFactory and AllTableMetricNameFactory can
create it instead of regular ObjectName to save memory.

It is possible to save memory because each name created by
TableMetricNameFactory (or AllTableMetricNameFactory) shares
most of its data with other names created by the same factory
and there's no need to create multiple copies.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-05-12 19:08:37 +02:00
Piotr Jastrzebski
48408dc6a3 Ensure regular ObjectName is returned to remote callers
Next patch will introduce new ObjectName implementation that
will use less memory. This new object won't be serializable.
This means it won't be possible to transport it to a remote
caller. We want to keep this new object local to JMX server as well.

This patch makes sure that every ObjectName returned
from APIBeanServer is transformed into a regular ObjectName.

It also makes sure that every ObjectInstance returned from
APIBeanServer has its ObjectName swapped with a regular ObjectName.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
2018-05-12 18:54:38 +02:00