Commit Graph

555 Commits

Author SHA1 Message Date
Calle Wilund
1219faf9f1 scylla-jmx: Fix TableMetricObjectName serialization
Fixes #133

TableMetricObjectName is not serializable as such, since
it depends on a lexicon object etc.

Use writeReplace to put a regular ObjectName in
the stream instead.
2020-09-01 15:46:18 +02:00
Amos Kong
d998ac2e1e install.sh: check both openjdk-8 and openjdk-11
On Debian10, only Openjdk-11 is available, the install.sh fails in java
checking. Openjdk-8 and Openjdk-11 all work well, we should check both of them.
This patch also fixed the error message.

Signed-off-by: Amos Kong <amos@scylladb.com>
2020-08-28 01:24:37 +08:00
Amos Kong
6311525346 nonroot.conf: set WorkingDirectory to empty
After offline installation, scylla-jmx fails to be started for a chdir
error. WorkingDirectory is set to /var/lib/scylla in scylla-jmx.service,
it doesn't exist in nonroot install. This patch solved the problem by
setting WorkingDirectory to empty in nonroot.conf.

$ systemctl --user status scylla-jmx
● scylla-jmx.service - Scylla JMX
   Loaded: loaded (/home/scylla-test/.config/systemd/user/../../../scylladb/etc/systemd/scylla-jmx.service; linked; vendor preset: enabled)
  Drop-In: /home/scylla-test/.config/systemd/user/scylla-jmx.service.d
           └─nonroot.conf
   Active: failed (Result: exit-code) since Wed 2020-08-26 15:19:56 UTC; 2s ago
  Process: 66955 ExecStart=/home/scylla-test/install_root/jmx/scylla-jmx $SCYLLA_JMX_PORT $SCYLLA_API_PORT $SCYLLA_API_ADDR $SCYLLA_JMX_ADDR $SCYLLA_JMX_FILE $SCYLLA_JMX_LOCAL $SCYLLA_JMX_REMOTE $SCYLLA_JMX_DEBUG (code=exited, status=200/CHDIR)
 Main PID: 66955 (code=exited, status=200/CHDIR)

systemd[5654]: Started Scylla JMX.
systemd[66955]: scylla-jmx.service: Changing to the requested working directory failed: No such file or directory
systemd[66955]: scylla-jmx.service: Failed at step CHDIR spawning /home/scylla-test/scylladb/jmx/scylla-jmx: No such file or directory
systemd[5654]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
systemd[5654]: scylla-jmx.service: Failed with result 'exit-code'.

Signed-off-by: Amos Kong <amos@scylladb.com>
2020-08-26 23:34:28 +08:00
Yaron Kaikov
d5d1efd188 dist/debian: Remove conflict tag for Java 11
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
2020-08-25 15:46:04 +03:00
Yaron Kaikov
23da40b559 dist/debian: Remove conflict tag for Java 11
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
2020-08-24 09:13:34 +03:00
Takuya ASADA
be8f1ac511 dist/common/systemd: set WorkingDirectory to get heap dump correctly
Currently scylla-jmx.service's PWD is "/", we get following error when
JVM trying to write heap dump on current directory:

Aug 17 05:52:15 localhost.localdomain scylla-jmx[3469]: Starting the JMX server
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: java.lang.OutOfMemoryError: Java heap space
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Dumping heap to java_pid3469.hprof ...
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Unable to create java_pid3469.hprof: Permission denied

To fix this, we need to specify WorkingDirectory on systemd unit.
2020-08-17 09:54:38 +03:00
Avi Kivity
c5ed83178a dist: debian: support non-x86
The package is already arch independent, so remove the artifical
restriction to x86.
2020-08-04 13:07:42 +03:00
Avi Kivity
626fd75173 dist: debian: do not require root during package build
Debian package builds provide a root environment for the installation
scripts, since that's what typical installation scripts expect. To
avoid providing actual root, a "fakeroot" system is used where syscalls
are intercepted and any effect that requires root (like chown) is emulated.

However, fakeroot sporadically fails for us, aborting the package build.
Since our install scripts don't really require root (when operating in
the --packaging mode), we can just tell dpkg-buildpackage that we don't
need fakeroot. This ought to fix the sporadic failures.

As a side effect, package builds are faster.

Follows scylla.git's b608af870b0a1ad88b91a72bddeff0c321877f9e.

Refs scylladb/scylla#6655.
2020-07-29 12:53:20 +03:00
Piotr Jastrzebski
c0d9d0f051 add build/ to gitignore
This directory is created by a build and shouldn't be commited so
it's best for git to just ignore it.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <b20dba12fb726aebd51b2ab9494e7c52f8058feb.1595259605.git.piotr@scylladb.com>
2020-07-21 09:10:25 +03:00
Pekka Enberg
7578d359af reloc: Add "--builddir" option to build_{rpm,deb}.sh
We need the ability to control build directory in scylla.git build
system. Let's add support for the "--builddir" option like in other
variants of the same scripts.

Message-Id: <20200717085723.701209-1-penberg@scylladb.com>
2020-07-18 12:16:31 +03:00
Avi Kivity
aa94fe53e0 dist: redhat: reduce log spam from unpacking sources when building rpm
rpmbuild defaults to logging the name of every file it unpacks from
the archive. This is quite a lot for Java applications.

Make it quiet with the %setup -q flag.
2020-07-15 12:27:43 +03:00
Pekka Enberg
4727910b5e Merge 'gitignore: fix typo and add scylla-apiclient/target/' from Benny
When building scylla-jmx in place, `scylla-apiclient/target/` is left
behind and should be ignored by `.gitignore` otherwise the scylla
submodule directory appears to be dirty.

* bhalevy-gitignore:
  gitignore: do not track scylla-apiclient/target/
  gitignore: fix typo in dependency-reduced-pom.xml
2020-07-14 10:21:44 +03:00
Pekka Enberg
15eb6adf92 apiclient: Bump Jackson version to 2.10.4
Jackson 2.9.x has various vulnerabilities that are fixed in 2.10 series:

https://github.com/FasterXML/jackson-databind/issues/2700#issuecomment-619590967

Let's update to the latest version of Jackson. This is a similar fix to
Github's Dependabot proposal, except we bump the version number across
all Jackson components:

https://github.com/scylladb/scylla-jmx/pull/116
2020-07-14 10:19:49 +03:00
Takuya ASADA
5820992a8e dist/debian: apply generated package version for .orig.tar.gz file
Same as scylladb/scylla#6752,
we currently does not able to apply version number fixup for .orig.tar.gz file,
even we applied correct fixup on debian/changelog, becuase it just reading
SCYLLA-VERSION-FILE.
We should parse debian/{changelog,control} instead.

Fixes #120
2020-07-06 12:49:15 +03:00
Benny Halevy
38eb871383 gitignore: do not track scylla-apiclient/target/
It is created when building jmx.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2020-06-24 13:59:34 +03:00
Benny Halevy
28fe33e588 gitignore: fix typo in dependency-reduced-pom.xml
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
2020-06-24 13:59:12 +03:00
Pekka Enberg
b2195734cc Upgrade to Guava 29.0
CVE-2018-10237 impacts Guava 24.1.0 and earlier, so let's upgrade to the latest version.

Reported-by: GitHub and Shlomi Livne
2020-06-16 10:04:48 +03:00
Juliusz Stasiewicz
b2e4796901 Added support for checkAndRepairCdcStreams command 2020-06-15 14:58:13 +03:00
Takuya ASADA
78c3b7627f dist/debian: cleanup build/debian before building .deb
On 52bd496, we stopped to rm -rf debian/ on build_deb.sh, since now we have
prebuilt debian/ directory.
However, it might cause .deb build error when we modified debian package source,
since it never cleanup.

To prevent build error, we need to cleanup build/debian on reloc/build_deb.sh,
before extracting contents from relocatable package.
2020-06-08 18:15:04 +03:00
Takuya ASADA
e0b21b9a19 dist: add --packaging for .rpm/.deb build
354df10 mistakenly does not contain dist/redhat & dist/debian change,
add --package option to them.
2020-06-08 17:45:52 +03:00
Takuya ASADA
2883a8dc63 dist/debian: don't install systemd unit by install.sh, use debian/*.service
Installing *.service by install.sh script causes the error on installing .deb
package, use debian/*.service instead.

Fixes scylladb/scylla#6010
Related scylladb/scylla#5640
Related 29285b28e2
2020-06-08 12:24:03 +03:00
Takuya ASADA
354df10ea9 install.sh: add dependency check and postinst script for manual install
To install scylla-jmx using install.sh easily, we need:
 - run dependency check before install
 - run postinst script after install

But we don't want to run them when we build .rpm/.deb package,
we also need to add --packaging option to skip them.

See scylladb/scylla#5830
2020-06-08 12:22:28 +03:00
Takuya ASADA
3fb777a8f0 dist/debian: support version number containing '_'
.deb packaging system does not support version number contains '_',
it should be replacedwith '-'
2020-06-04 05:27:04 +09:00
Takuya ASADA
f044c8988e dist/debian: move version number fixup to debian_files_gen.py
Now we generate dist/changelog on relocatable package generation time,
we cannot run '.rc' fixup on .deb package building time, need to do it
in debian_files_gen.py.
2020-06-04 05:27:02 +09:00
Takuya ASADA
ec2a830876 reloc-pkg: move all files under project name directory
To make unified relocatable package easily, we may want to merge tarballs to single tarball like this:
zcat .tar.gz | gzip -c > scylla-unified.tar.xz
But it's not possible with current relocatable package format, since there are multiple files conflicts, install.sh, SCYLLA--FILE, dist/, README.md, etc..

To support this, we need to archive everything in the directory when building relocatable package.

See: scylladb/scylla#6315
2020-06-03 09:53:11 +03:00
Amnon Heiman
9628cc0728 StorageService: Add the scrub 3.11 command implementation
The scrub command was not supported from node_tool, but now when we want
to enable it the current API is not compatible with the 3.11 MBean
definition.

This patch adds the definition to the MBean and the implementation to
StorageService.

It also address two problems with the old scrub implementation, just
in case someone will use them.

1. Implementation didn't pass the parameters to the API.
2. A stub implementation called itself instead of calling an actual
implementation.

This patch will enable to test the command from nodetool additional
changes may come on top of it if more command line options will be
supported.

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2020-05-29 14:12:09 +03:00
Ivan Prisyazhnyy
c7dcbd7f42 fix is auto compaction disabled
align API to the recent changes at https://github.com/scylladb/scylla/pull/6176

don't wrap API exceptions into IOException for enableAutoCompaction
2020-05-29 14:02:40 +03:00
dependabot[bot]
fc43c56369 build(deps): bump jackson-databind in /scylla-apiclient
Bumps [jackson-databind](https://github.com/FasterXML/jackson) from 2.9.10.1 to 2.9.10.4.
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

Signed-off-by: dependabot[bot] <support@github.com>
2020-05-29 14:00:31 +03:00
Takuya ASADA
52bd496006 dist/debian: drop dependency on pystache
Drop dependency on pystache since it nolong present in Fedora 32.

To implement it, simplified debian package build process.
It will be generate debian/ directory when building relocatable package,
we just need to run debuild using the package.

To generate debian/ directory this commit added debian_files_gen.py,
it construct whole directory including control and changelog files
from template files.
Since we need to stop pystache, these template files swiched to
string.Template class which is included python3 standard library.

see: https://github.com/scylladb/scylla/pull/6313
2020-05-23 06:08:01 +03:00
Takuya ASADA
18f8acc60e dist/redhat: drop dependency on pystache
Same as https://github.com/scylladb/scylla/pull/6313, drop dependency on
pystache since it nolong present in Fedora 32.
2020-05-19 08:15:59 +03:00
Takuya ASADA
773a82d539 dist: allow specify JVM options from sysconfig (#93)
Add SCYLLA_JMX_JVM_OPTS on sysconfig to specify JVM options.

Reviewed-by: Ľuboš Koščo <lubos@scylladb.com>

Fixes #58
2020-01-28 12:43:03 +02:00
Takuya ASADA
46681753cd dist: add /usr/lib/scylla/jmx for compatibility (#91)
On the commit 4c8660d, we dropped /usr/lib/scylla/jmx since it likely no user
script invoke scripts under the directory.
However, we found there are possibility scylla-jmx.service tries to load .jar
file from /usr/lib/scylla/jmx, when user upgraded from older version of scylla.
Because /etc/sysconfig/scylla-jmx is marked as 'noreplace' on our rpm,
yum upgrade may keep old sysconfig file when it modified by user, that may
causes to load .jar from /usr/lib/scylla/jmx since we specify the path in the
sysconfig file.

To avoid the issue it's better to have symlinks on /usr/lib/scylla/jmx for
safety.

See #90
2020-01-16 15:51:39 +02:00
Takuya ASADA
29601254fc dist/redhat: call systemctl --daemon-reload when upgraded (#92)
Since %systemd_post does not call systemctl --daemon-reload, we need to call it
manually to apply changes.

Fixes #90
2020-01-08 13:35:38 +02:00
Takuya ASADA
4c8660d41a dist: drop symlink to scripts (#89)
This is scylla-jmx part of https://github.com/scylladb/scylla/pull/5530

After we stopped replacing /usr/lib/scylla with symlink,
creating symlink on /opt/scylladb/scripts/jmx become meaningless,
so we can drop it now.

We able to introduce new symlink on /usr/lib/scylla, but it likely no user
directly invoke scripts under /usr/lib/scylla/jmx, so we don't have to
do that.
2019-12-30 13:51:33 +02:00
Takuya ASADA
2f34a97c6e dist/debian: add AdoptOpenJDK for Debian 10
Since Debian 10 dropped OpenJDK 8, so we need to switch to other
JVM distribution which still maintain OpenJDK 8 for Debian.
We can use AdoptOpenJDK for such environment.

See scylladb/scylla#5463

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20191218124838.35017-1-syuu@scylladb.com>
2019-12-18 15:32:04 +02:00
Takuya ASADA
236ffa6c98 dist/debian: add Conflicts with openjdk-11
Since Debian variants can install multiple JRE, "Depends: openjdk-8-jre" may
not mean default JRE is openjdk-8.
We should block openjdk-11 for now, until we support it.

Related with scylladb/scylla#5463

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20191218111835.25618-1-syuu@scylladb.com>
2019-12-18 14:08:28 +02:00
Takuya ASADA
771cf6ea50 dist/redhat: force xz compression on rpm binary payload
Same as 301c835cbf,
Fedora 31 switched the default compression to zstd, which isn't readable
by some older rpm distributions (CentOS 7 in particular). Tell it to use
the older xz compression instead, so packages produced on Fedora 31 can
be installed on older distributions.

See: https://github.com/scylladb/scylla/pull/5310

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20191210191441.108774-1-syuu@scylladb.com>
2019-12-10 22:18:29 +02:00
Takuya ASADA
31e6bcf9be dist/redhat: fix rpmbuild error on Fedora 31
Same as scylladb/scylla-ami#53, it seems like rpm macro %systemd_postun requires
 one argument starting from Fedora 31, otherwise it causes the error.
The solution is passing systemd unit name just like
%systemd_post and %systemd_preun.

see: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/TU3T2ZYY67SMAJFR2TD4HY6SCPPDVS5V/

Fixes #87

Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <20191205120514.9382-1-syuu@scylladb.com>
2019-12-05 17:03:10 +02:00
Alexandros Bantis
d8c47603d9 Create a HTTP client per instance (#86)
Create javax HTTP client once per instance instead of per request.

Fixes #82
2019-11-19 17:28:09 +02:00
Pekka Enberg
f45ae1833e Add Pekka as a "code owner" on GitHub (#85)
Add myself as a "code owner" so that I am assigned a review
automatically:

https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners

I also wanted to add Amnon and Calle, but apparently you need to have
write permissions in order to be a code owner.

The purpose of this automation is to ensure Scylla JMX pull requests
show up in my github.com/pulls page. Thanks Maciej Zimnoch for the tip!
2019-11-14 02:18:33 -08:00
dependabot[bot]
8e1beb11f4 Upgrade jackson-databind from 2.9.9 to 2.9.10.1 (#84)
This upgrades jackson-databind dependency from version 2.9.9 to 2.9.10.1, which fixes various security vulnerabilities:

https://www.cvedetails.com/vulnerability-list/vendor_id-15866/product_id-42991/Fasterxml-Jackson-databind.html
2019-11-13 19:57:55 +02:00
Glauber Costa
27fed6136a Run scylla-jmx in a systemd slice (#79)
Scylla now supports server-defined systemd slices that are used to provide
isolation between components.

This patch adds scylla-jmx to the helper slice. This will guarantee that
scylla-jmx does not use too much resources, influencing the server performance.

Signed-off-by: Glauber Costa <glauber@scylladb.com>
2019-11-13 18:28:14 +02:00
Amos Kong
fa00e84794 build_deb.sh: don't generate scylla-jmx.service from mustache template (#81)
Takuya had untemplataize scylla-jmx.service in commit e8355087ea
But the build_deb.sh still tries to generate service file from a deleted
mustache template file -- scylla-jmx.spec.mustache. It wrongly redirects
a path to service file, then scylla-jmx would fail to start.

Fixes #80
2019-10-01 18:35:50 +03:00
Calle Wilund
f915f8fc7a sstableinfo: Fix deserizalization of "properties"
Refs #76

Since the incoming json uses swagger "key", "value" syntax
we need to do explicit deserialization of this property
as well (not just extended props).

Message-Id: <20190930115432.27801-1-calle@scylladb.com>
2019-09-30 15:29:52 +03:00
Avi Kivity
dc7f37b901 Merge "nonroot installer" from Takuya
"This is nonroot installer patchset v4, for scylla-jmx."

* 'nonroot_v4' of https://github.com/syuu1228/scylla-jmx:
  install.sh: add --nonroot mode
  dist/common/systemd: untemplataize *.service, use drop-in units instead
  dist: move package build script to install.sh
2019-09-10 14:46:04 +03:00
Takuya ASADA
9ef12f4651 install.sh: add --nonroot mode
This implements the way to install Scylla without requires root privilege,
not distribution dependent, does not uses package manager.
2019-09-04 09:54:42 +09:00
Takuya ASADA
e8355087ea dist/common/systemd: untemplataize *.service, use drop-in units instead
Since systemd unit can override parameters using drop-in unit, we don't need
mustache template for them.

Also, drop --disttype option on install.sh since it does not required anymore,
introduce --sysconfdir instead for non-redhat distributions.
2019-09-04 08:54:05 +09:00
Takuya ASADA
a1044e3bd1 dist: move package build script to install.sh
Move package build script from .rpm/.deb to single script, install.sh.
We need this to support nonroot mode, and also to keep package build script
consistent between .rpm/.deb.
2019-09-04 08:30:37 +09:00
Pekka Enberg
04ea3ab7e0 Merge 'Implement sstable_info command' from Calle
"Fixes #76

Implements JMX level call for "sstable_info" REST api command.

Requires seastar patch:
json: Make date formatter use RFC8601/RFC3339 format

Requires scylla patch set "Implement sstable_info API command (info on sstables)"

Forwards call to REST sstable_info and packs the data
into CompositeData for JMX consumption."
* 'sstabledesc' of git://github.com/elcallio/scylla-jmx:
  storage_service: Add "getSSTableInfo" command/attribute
  service: Add objects for deserializing sstable_info json
  scylla-apiclient: Add Date json serializer helper
  APIClient: Add jackson JSON serializer support to client object
  apiclient/pom.xml: Add jackson JSON support libs for REST client
2019-08-13 14:40:25 +03:00
Calle Wilund
133b2e4728 storage_service: Add "getSSTableInfo" command/attribute
Fixes #76

Requires seastar patch:
 json: Make date formatter use RFC8601/RFC3339 format

Requires scylla patch set "Sstabledesc"

Forwards call to REST sstable_info and packs the data
into CompositeData for JMX consumption.
2019-08-06 08:12:14 +00:00