Commit Graph

546 Commits

Author SHA1 Message Date
Avi Kivity
26a6919714 build: replace yum with dnf
dnf has replaced yum on Fedora and CentOS. On modern versions of Fedora,
you have to install an extra package to get the old name working, so
avoid that inconvenience and use dnf directly.

Closes #181
2021-11-15 15:19:22 +02:00
Avi Kivity
d6225c5231 build: use utc for build datestamp
This helps keep packages built on different machines have the
same datestamp, if started on the same time.
2021-11-07 15:58:09 +02:00
Benny Halevy
48d37f3402 StorageService: scrub: fix scrubMode is empty condition
`!=` compares references not values.

Use !"".equals(scrubMode) instead, as it also covers
the null scrubMode case.

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #179
2021-11-02 15:21:08 +02:00
Takuya ASADA
5c383b641b reloc: stop removing entire $BUILDDIR
We found that user can mistakenly break system with --builddir option,
something like './reloc/build_deb.sh --builddir /'.
To avoid that we need to stop removing entire $BUILDDIR, remove
directories only we have to clean up before building deb package.

See: https://github.com/scylladb/scylla-python3/pull/23#discussion_r707088453

Closes #177
2021-09-19 10:01:40 +03:00
Juliusz Stasiewicz
658818b2d0 Support --load-and-stream option from nodetool refresh
This information is translated to {"load_and_stream", "true"} entry in the
POST request to Scylla's HTTP API at `storage_service/sstables/{keyspace}`
endpoint.

More about this feature: scylladb/scylla#7846

This change is a consequence of scylladb/scylla-tools-java#253.
2021-09-13 18:22:19 +03:00
Benny Halevy
70b19e6270 scrub: support scrubMode and deprecate skipCorrupted
Support new scrubMode option and deprecate skipCorrupted
that's equivalent to scrubMode="SKIP".

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #175
2021-08-24 14:51:05 +03:00
Benny Halevy
5311e9bae3 storage_service: takeSnapshot: support the skipFlush option
Fixes #167

Signed-off-by: Benny Halevy <bhalevy@scylladb.com>

Closes #168
2021-06-18 12:58:27 +03:00
dependabot[bot]
fbfbdaa298 build(deps): bump snakeyaml from 1.16 to 1.26 in /scylla-apiclient
Bumps [snakeyaml](https://bitbucket.org/asomov/snakeyaml) from 1.16 to 1.26.
- [Commits](https://bitbucket.org/asomov/snakeyaml/branches/compare/snakeyaml-1.26..v1.16)

---
updated-dependencies:
- dependency-name: org.yaml:snakeyaml
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

Closes #169
2021-06-10 09:59:48 +03:00
Piotr Wojtczak
a7c4c39dd0 storage_service: Fix getToppartitions to always return both reads and writes
In line with the previous API, the getToppartitions function returned
results for one specified sampler (reads OR writes). This forced
the user to call the function once for each sampler, which is
suboptimal.
This commit changes the signature so that results for both samplers
are returned and the user can then pick whichever they need.
2021-05-10 18:07:07 +03:00
Piotr Wojtczak
440313eb72 storage_service: Add a generic toppartitions endpoint
As part of making the toppartitions API more generic
(i.e. being able to consider multiple tables
and keyspaces specified by the user) this commit adds
a JMX endpoint to call the generic Scylla REST API
introduced in #7864. It has been put inside
storage_service as being now able to query more than
one column family makes it no longer suitable for the
'column_family' group.

Fixes #4520
2021-03-25 12:35:18 +02:00
Takuya ASADA
9c687b562e dist/redhat: add support SLES
CentOS/RHEL and SLES has differnt package name of opejdk, use common
name of JRE.
Note that using common name of Java package is also useful when user want to
use differnt implementation of JRE for Scylla.

Also, disable AutoReqProv which is mistakenly enabled but required for
cross build rpm.
2021-03-15 17:23:14 +02:00
Amnon Heiman
15c1d4f43f StorageService: Add a method to return the uptime
Currently, the nodetool uses the jmx server for the uptime, this is
confusing is what we expect is Scylla uptime.

This patch exposes the API uptime using MBean.

Relates to #154

Signed-off-by: Amnon Heiman <amnon@scylladb.com>

Closes #155
2021-03-04 10:52:08 +02:00
dependabot[bot]
ffab41d714 Bump Jackson version in scylla-apiclient
Bumps `jackson.version` from 2.10.4 to 2.12.1.

Updates `jackson-annotations` from 2.10.4 to 2.12.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

Updates `jackson-databind` from 2.10.4 to 2.12.1
- [Release notes](https://github.com/FasterXML/jackson/releases)
- [Commits](https://github.com/FasterXML/jackson/commits)

Updates `jackson-jaxrs-json-provider` from 2.10.4 to 2.12.1

Signed-off-by: dependabot[bot] <support@github.com>

Closes #159
2021-03-04 10:48:34 +02:00
Pekka Enberg
bac7d0b31e Merge 'Fix locking in APIBuilder.remove()' from Pekka Enberg
This pull request reverts the commit c2fc96b ("APIBuilder: Remove
RW-lock in JMX server repository wrapper") and fixes a missing unlock
from APIBuilder.remove().

Closes #163

* github.com:scylladb/scylla-jmx:
  APIBuilder: Unlock RW-lock in remove()
  Revert "APIBuilder: Remove RW-lock in JMX server repository wrapper"
2021-03-03 18:29:57 +02:00
Pekka Enberg
59fd4d2b03 APIBuilder: Unlock RW-lock in remove()
The remove() function accidentally calls lock() in the finally
block, leaving the RW-lock unlocked.

Refs: scylladb/scylla#7991
2021-03-03 18:23:41 +02:00
Pekka Enberg
9d7ee8af3c Revert "APIBuilder: Remove RW-lock in JMX server repository wrapper"
This reverts commit c2fc96be71. The
RW-lock usage had a bug, which will be fixed in a follow up patch.
2021-03-03 18:20:46 +02:00
Calle Wilund
c2fc96be71 APIBuilder: Remove RW-lock in JMX server repository wrapper
This is a seemingly pointless change. The RW-lock code is 100%
correct (afaict), yet we've seen repeated cases of test runs
hanging in JMX query because this lock is seemingly left held
by what seems to be the reaper task.

There is no explanation for this, no sign of exceptions/errors
that could explain the lock being broken. Nor any known JDK/JVM
bugs.

Yet, in tests, it seems that replacing the lock with a more
coarse, yet proven, synchronized, fixes the issue. So there.

I officially hate this patch, and it should not exist.
2021-03-03 15:40:33 +02:00
Amnon Heiman
8073af6e06 CompactionManager: add the compaction id when available
This patch adds the compaction id in getCompactions if it returns by the
API, if it's not the current behaviour will be used and it will return none.

After this patch a call to nodetool compactionstats -H

Will return:

id                                   compaction type keyspace  table     completed total unit progress
c942bd30-7a62-11eb-84bc-576502584f9a COMPACTION      keyspace1 standard1 1062      8576  keys 12.38%
c9429620-7a62-11eb-8afb-576402584f9a COMPACTION      keyspace1 standard1 972       8448  keys 11.51%
Active compaction remaining time :   0h00m00s

Fixes scylladb/scylla#7927

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2021-03-01 10:04:08 +02:00
Pekka Enberg
bf8bb16b52 Merge 'dist/debian: fix renaming debian/scylla-* files rule' from Takuya ASADA
Current renaming rule of debian/scylla-* files is buggy, it fails to
install some .service files when custom product name specified.

Introduce regex based rewriting instead of adhoc renaming, and fixed
wrong renaming rule.

Fixes scylladb/scylla#8113

Closes #158

* github.com:scylladb/scylla-jmx:
  dist/debian: fix renaming debian/scylla-* files rule
  dist/debian: sync packaging script with scylla main repo
2021-02-18 10:34:29 +02:00
Takuya ASADA
8f62d71e11 dist/debian: fix renaming debian/scylla-* files rule
Current renaming rule of debian/scylla-* files is buggy, it fails to
install some .service files when custom product name specified.

Introduce regex based rewriting instead of adhoc renaming, and fixed
wrong renaming rule.

Related scylladb/scylla#8113
2021-02-18 04:38:36 +09:00
Takuya ASADA
3618481e23 dist/debian: sync packaging script with scylla main repo 2021-02-18 04:25:48 +09:00
Takuya ASADA
949cefc251 dist/redhat: stop using systemd macros, call systemctl directly
Fedora version of systemd macros does not work correctly on CentOS7,
since CentOS7 does not support "file trigger" feature.
Even after 05d4378, scriptlets on old scylla .rpm and new scylla .rpm is
not completely same.

To fix the issue we need to stop using systemd macros, call systemctl
directly.

Fixes #94
2021-02-02 11:28:55 +02:00
Piotr Wojtczak
611d586981 Remove obsolete FIXME
The cardinality problem has already been fixed in #149.
2021-01-25 13:07:40 +02:00
Amos Kong
2c9565024f install.sh: set a valid WorkingDirectory for nonroot offline install
In commit 6311525, we set an empty value to WorkDirectory for nonroot.conf
of scylla-jmx.service. It works with ubuntu16, debian9, debian 10. But it
doesn't work with ubuntu 18.

This patch changed the WorkingDirectory of nonroot offline install to
default install directory (/home/scylla-test/scylladb).

Fixes #151

Signed-off-by: Amos Kong <amos@scylladb.com>
2020-12-28 21:18:35 +02:00
Piotr Wojtczak
20469bf749 column_family: Return proper cardinality for toppartitions requests
Right now, in the finishLocalSampling method of the ColumnFamilyStore
we return the size of the list of returned partitions. Instead, we should
be propagating the actual cardinality of the sampled set.
Let's just read the read_cardinality and write_cardinality properties
of the scylla's REST API response.

Fixes #148
2020-12-13 13:50:56 +02:00
Eliran Sinvani
6174a47924 Relocatable Package: create product prefixed relocatable archive
The build system was hardcoded to produce a package that is
prefixed with scylla instead of the product name. This is not
in line with out CI system requirements and can be also a source
for confusion.
This commit make the packaging system generate a package of
the format: {product}-jmx-package.tar.gz instead of
scylla-jmx-package.tar.gz

Closes #146
2020-10-15 17:10:21 +03:00
dependabot[bot]
91e7adffb1 build(deps-dev): bump junit from 4.8.2 to 4.13.1
Bumps [junit](https://github.com/junit-team/junit4) from 4.8.2 to 4.13.1.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.13.1.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.8.2...r4.13.1)

Signed-off-by: dependabot[bot] <support@github.com>
2020-10-15 14:22:38 +03:00
Amnon Heiman
c51906ed01 StorageService.java: Use the endpoint for getRangeToEndpointMap
After implementing range_to_endpoint_map endpoint update the API call to
it.

Fixes #36

Signed-off-by: Amnon Heiman <amnon@scylladb.com>
2020-10-08 11:53:33 +03:00
Takuya ASADA
c55f3f292b dist/debian/debian_files_gen.py: don't ignore permission error on shutil.rmtree()
shutil.rmtree(ignore_errors=True) was for ignores error when directory not exist,
but it also ignores permission error, so we shouldn't use that.
Run os.path.exists() before shutil.rmtree() instead.

See scylladb/scylla#7337
2020-10-08 11:48:30 +03:00
Takuya ASADA
e3a381d5a1 install.sh: show warning nonroot mode when systemd does not support user mode
On older distribution such as CentOS7, it does not support systemd user mode.
On such distribution nonroot mode does not work, show warning message and
skip running systemctl --user.

See scylladb/scylla#7071
2020-10-07 11:33:22 +03:00
Takuya ASADA
25bcd76017 install.sh: stop using symlinks for systemd units on nonroot mode
On some environment, systemctl enable <service> fails when we use symlink.
So just directly copy systemd units to ~/.config/systemd/user, instead of
creating symlink.

See scylladb/scylla#7288
2020-09-29 13:31:30 +03:00
Avi Kivity
45e4f28766 build: support passing product-version-release as a parameter
Instead of using the baked-in values from SCYLLA-VERSION-GEN,
allow passing an override. This will be used by the supermodule
to have an identical product-version-release (especially release,
which contains the git hash) across all packages.
2020-09-23 12:57:50 +03:00
Avi Kivity
6795a22afe Merge "dist: do not install build dependencies on build script" from Takuya
"
We do not want to install dependencies on package building time,
we want to install it on dbuild container.
So drop package installation from the scripts.

scylladb/scylla#7219
"

Closes #138.

* 'dont_install_builddep' of https://github.com/syuu1228/scylla-jmx:
  reloc: simplified .deb build process
  reloc: simplified .rpm build process
  dist: do not install build dependencies on build script
  dist/debian: Remove conflict tag for Java 11
2020-09-16 10:29:23 +03:00
Takuya ASADA
5fa422c4ea reloc: simplified .deb build process
We don't really need to have two build_deb.sh, merge it to reloc.
2020-09-16 16:19:24 +09:00
Takuya ASADA
f1612ef508 reloc: simplified .rpm build process
We don't really need to have two build_rpm.sh, merge it to reloc.
2020-09-16 16:19:24 +09:00
Pekka Enberg
d3096f32e0 dist: debian: fix detection of debuild
Fix the path of debuild, similar to what commit
f57fbb77b0a3f8d240c9924f3fa4529f5b5c8122 ("dist: debian: fix detection
of debuild") did in scylla-tools.

Message-Id: <20200911164153.34699-1-penberg@scylladb.com>
2020-09-13 16:26:22 +03:00
Takuya ASADA
99e491df40 dist: do not install build dependencies on build script
We do not want to install dependencies on package building time,
we want to install it on dbuild container.
So drop package installation from the scripts.

scylladb/scylla#7219
2020-09-13 20:19:16 +09:00
Pekka Enberg
8d92e5450e Merge 'JMX footprint work' from Calle
"
Fixes #133
Fixes #134
Refs #135

Makes CF mbean refresh code synchronized and tries to remove reductant
calls if we contend. Adds background reaping of dead objects to reduce
memory load in (test) scenarios where we manage to refresh to add, but
not cause removal (i.e. no wildcard queries).

TableMetricsObjectName serialization is fixed in the series because
without it we see loads of exceptions when refreshing the mbean set.
"

* elcallio-jmx-fixes:
  scylla-jmx: Use registration checker objects
  scylla-jmx: Introduce a registration check object
  scylla-jmx: Fix TableMetricObjectName serialization
2020-09-07 13:54:56 +03:00
Calle Wilund
ba3f58c63c scylla-jmx: Use registration checker objects
Fixes #134
Refs #135

Replaces previous refresh calls with ones bound to registration
check objects, which provides some sync between threads doing
refresh, and reduced redundant calls.

Also adds repeated reaping of dead objects, i.e. every 5 minutes
we try to remove dead CF:s (not adding new ones), to reduce
idle footprint.
2020-09-07 11:00:42 +02:00
Calle Wilund
771fe3e360 scylla-jmx: Introduce a registration check object
Allows for shared code for synchronized and optionally
partial update checks.
2020-09-07 11:00:42 +02:00
Pekka Enberg
12ab6aaeb8 Merge "Fix JMX startup after offline installation" from Amos
"Currently after offline installation, the scylla-jmx fails to start.
 This pull request fixes issues with openjdk version detection and
 working directory configuration to make scylla-jmx start.

 Fixes: scylladb/scylla#7098 by [PATCH] install.sh: check both openjdk-8 and openjdk-11

 Fixes: #129 by [PATCH] nonroot.conf: set WorkingDirectory to empty"

Reviewed-by: Takuya ASADA <syuu@scylladb.com>
* 'openjdk' of git://github.com/amoskong/scylla-jmx:
  install.sh: check both openjdk-8 and openjdk-11
  nonroot.conf: set WorkingDirectory to empty
2020-09-07 09:38:28 +03:00
Calle Wilund
1219faf9f1 scylla-jmx: Fix TableMetricObjectName serialization
Fixes #133

TableMetricObjectName is not serializable as such, since
it depends on a lexicon object etc.

Use writeReplace to put a regular ObjectName in
the stream instead.
2020-09-01 15:46:18 +02:00
Amos Kong
d998ac2e1e install.sh: check both openjdk-8 and openjdk-11
On Debian10, only Openjdk-11 is available, the install.sh fails in java
checking. Openjdk-8 and Openjdk-11 all work well, we should check both of them.
This patch also fixed the error message.

Signed-off-by: Amos Kong <amos@scylladb.com>
2020-08-28 01:24:37 +08:00
Amos Kong
6311525346 nonroot.conf: set WorkingDirectory to empty
After offline installation, scylla-jmx fails to be started for a chdir
error. WorkingDirectory is set to /var/lib/scylla in scylla-jmx.service,
it doesn't exist in nonroot install. This patch solved the problem by
setting WorkingDirectory to empty in nonroot.conf.

$ systemctl --user status scylla-jmx
● scylla-jmx.service - Scylla JMX
   Loaded: loaded (/home/scylla-test/.config/systemd/user/../../../scylladb/etc/systemd/scylla-jmx.service; linked; vendor preset: enabled)
  Drop-In: /home/scylla-test/.config/systemd/user/scylla-jmx.service.d
           └─nonroot.conf
   Active: failed (Result: exit-code) since Wed 2020-08-26 15:19:56 UTC; 2s ago
  Process: 66955 ExecStart=/home/scylla-test/install_root/jmx/scylla-jmx $SCYLLA_JMX_PORT $SCYLLA_API_PORT $SCYLLA_API_ADDR $SCYLLA_JMX_ADDR $SCYLLA_JMX_FILE $SCYLLA_JMX_LOCAL $SCYLLA_JMX_REMOTE $SCYLLA_JMX_DEBUG (code=exited, status=200/CHDIR)
 Main PID: 66955 (code=exited, status=200/CHDIR)

systemd[5654]: Started Scylla JMX.
systemd[66955]: scylla-jmx.service: Changing to the requested working directory failed: No such file or directory
systemd[66955]: scylla-jmx.service: Failed at step CHDIR spawning /home/scylla-test/scylladb/jmx/scylla-jmx: No such file or directory
systemd[5654]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
systemd[5654]: scylla-jmx.service: Failed with result 'exit-code'.

Signed-off-by: Amos Kong <amos@scylladb.com>
2020-08-26 23:34:28 +08:00
Yaron Kaikov
d5d1efd188 dist/debian: Remove conflict tag for Java 11
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
2020-08-25 15:46:04 +03:00
Yaron Kaikov
23da40b559 dist/debian: Remove conflict tag for Java 11
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
2020-08-24 09:13:34 +03:00
Takuya ASADA
be8f1ac511 dist/common/systemd: set WorkingDirectory to get heap dump correctly
Currently scylla-jmx.service's PWD is "/", we get following error when
JVM trying to write heap dump on current directory:

Aug 17 05:52:15 localhost.localdomain scylla-jmx[3469]: Starting the JMX server
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: java.lang.OutOfMemoryError: Java heap space
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Dumping heap to java_pid3469.hprof ...
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Unable to create java_pid3469.hprof: Permission denied

To fix this, we need to specify WorkingDirectory on systemd unit.
2020-08-17 09:54:38 +03:00
Avi Kivity
c5ed83178a dist: debian: support non-x86
The package is already arch independent, so remove the artifical
restriction to x86.
2020-08-04 13:07:42 +03:00
Avi Kivity
626fd75173 dist: debian: do not require root during package build
Debian package builds provide a root environment for the installation
scripts, since that's what typical installation scripts expect. To
avoid providing actual root, a "fakeroot" system is used where syscalls
are intercepted and any effect that requires root (like chown) is emulated.

However, fakeroot sporadically fails for us, aborting the package build.
Since our install scripts don't really require root (when operating in
the --packaging mode), we can just tell dpkg-buildpackage that we don't
need fakeroot. This ought to fix the sporadic failures.

As a side effect, package builds are faster.

Follows scylla.git's b608af870b0a1ad88b91a72bddeff0c321877f9e.

Refs scylladb/scylla#6655.
2020-07-29 12:53:20 +03:00
Piotr Jastrzebski
c0d9d0f051 add build/ to gitignore
This directory is created by a build and shouldn't be commited so
it's best for git to just ignore it.

Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <b20dba12fb726aebd51b2ab9494e7c52f8058feb.1595259605.git.piotr@scylladb.com>
2020-07-21 09:10:25 +03:00