We found that user can mistakenly break system with --builddir option,
something like './reloc/build_deb.sh --builddir /'.
To avoid that we need to stop removing entire $BUILDDIR, remove
directories only we have to clean up before building deb package.
See: https://github.com/scylladb/scylla-python3/pull/23#discussion_r707088453Closes#177
This information is translated to {"load_and_stream", "true"} entry in the
POST request to Scylla's HTTP API at `storage_service/sstables/{keyspace}`
endpoint.
More about this feature: scylladb/scylla#7846
This change is a consequence of scylladb/scylla-tools-java#253.
Support new scrubMode option and deprecate skipCorrupted
that's equivalent to scrubMode="SKIP".
Signed-off-by: Benny Halevy <bhalevy@scylladb.com>
Closes#175
In line with the previous API, the getToppartitions function returned
results for one specified sampler (reads OR writes). This forced
the user to call the function once for each sampler, which is
suboptimal.
This commit changes the signature so that results for both samplers
are returned and the user can then pick whichever they need.
As part of making the toppartitions API more generic
(i.e. being able to consider multiple tables
and keyspaces specified by the user) this commit adds
a JMX endpoint to call the generic Scylla REST API
introduced in #7864. It has been put inside
storage_service as being now able to query more than
one column family makes it no longer suitable for the
'column_family' group.
Fixes#4520
CentOS/RHEL and SLES has differnt package name of opejdk, use common
name of JRE.
Note that using common name of Java package is also useful when user want to
use differnt implementation of JRE for Scylla.
Also, disable AutoReqProv which is mistakenly enabled but required for
cross build rpm.
Currently, the nodetool uses the jmx server for the uptime, this is
confusing is what we expect is Scylla uptime.
This patch exposes the API uptime using MBean.
Relates to #154
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Closes#155
This pull request reverts the commit c2fc96b ("APIBuilder: Remove
RW-lock in JMX server repository wrapper") and fixes a missing unlock
from APIBuilder.remove().
Closes#163
* github.com:scylladb/scylla-jmx:
APIBuilder: Unlock RW-lock in remove()
Revert "APIBuilder: Remove RW-lock in JMX server repository wrapper"
This is a seemingly pointless change. The RW-lock code is 100%
correct (afaict), yet we've seen repeated cases of test runs
hanging in JMX query because this lock is seemingly left held
by what seems to be the reaper task.
There is no explanation for this, no sign of exceptions/errors
that could explain the lock being broken. Nor any known JDK/JVM
bugs.
Yet, in tests, it seems that replacing the lock with a more
coarse, yet proven, synchronized, fixes the issue. So there.
I officially hate this patch, and it should not exist.
This patch adds the compaction id in getCompactions if it returns by the
API, if it's not the current behaviour will be used and it will return none.
After this patch a call to nodetool compactionstats -H
Will return:
id compaction type keyspace table completed total unit progress
c942bd30-7a62-11eb-84bc-576502584f9a COMPACTION keyspace1 standard1 1062 8576 keys 12.38%
c9429620-7a62-11eb-8afb-576402584f9a COMPACTION keyspace1 standard1 972 8448 keys 11.51%
Active compaction remaining time : 0h00m00s
Fixesscylladb/scylla#7927
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Current renaming rule of debian/scylla-* files is buggy, it fails to
install some .service files when custom product name specified.
Introduce regex based rewriting instead of adhoc renaming, and fixed
wrong renaming rule.
Fixes scylladb/scylla#8113
Closes#158
* github.com:scylladb/scylla-jmx:
dist/debian: fix renaming debian/scylla-* files rule
dist/debian: sync packaging script with scylla main repo
Current renaming rule of debian/scylla-* files is buggy, it fails to
install some .service files when custom product name specified.
Introduce regex based rewriting instead of adhoc renaming, and fixed
wrong renaming rule.
Related scylladb/scylla#8113
Fedora version of systemd macros does not work correctly on CentOS7,
since CentOS7 does not support "file trigger" feature.
Even after 05d4378, scriptlets on old scylla .rpm and new scylla .rpm is
not completely same.
To fix the issue we need to stop using systemd macros, call systemctl
directly.
Fixes#94
In commit 6311525, we set an empty value to WorkDirectory for nonroot.conf
of scylla-jmx.service. It works with ubuntu16, debian9, debian 10. But it
doesn't work with ubuntu 18.
This patch changed the WorkingDirectory of nonroot offline install to
default install directory (/home/scylla-test/scylladb).
Fixes#151
Signed-off-by: Amos Kong <amos@scylladb.com>
Right now, in the finishLocalSampling method of the ColumnFamilyStore
we return the size of the list of returned partitions. Instead, we should
be propagating the actual cardinality of the sampled set.
Let's just read the read_cardinality and write_cardinality properties
of the scylla's REST API response.
Fixes#148
The build system was hardcoded to produce a package that is
prefixed with scylla instead of the product name. This is not
in line with out CI system requirements and can be also a source
for confusion.
This commit make the packaging system generate a package of
the format: {product}-jmx-package.tar.gz instead of
scylla-jmx-package.tar.gz
Closes#146
shutil.rmtree(ignore_errors=True) was for ignores error when directory not exist,
but it also ignores permission error, so we shouldn't use that.
Run os.path.exists() before shutil.rmtree() instead.
See scylladb/scylla#7337
On older distribution such as CentOS7, it does not support systemd user mode.
On such distribution nonroot mode does not work, show warning message and
skip running systemctl --user.
See scylladb/scylla#7071
On some environment, systemctl enable <service> fails when we use symlink.
So just directly copy systemd units to ~/.config/systemd/user, instead of
creating symlink.
See scylladb/scylla#7288
Instead of using the baked-in values from SCYLLA-VERSION-GEN,
allow passing an override. This will be used by the supermodule
to have an identical product-version-release (especially release,
which contains the git hash) across all packages.
"
We do not want to install dependencies on package building time,
we want to install it on dbuild container.
So drop package installation from the scripts.
scylladb/scylla#7219
"
Closes#138.
* 'dont_install_builddep' of https://github.com/syuu1228/scylla-jmx:
reloc: simplified .deb build process
reloc: simplified .rpm build process
dist: do not install build dependencies on build script
dist/debian: Remove conflict tag for Java 11
Fix the path of debuild, similar to what commit
f57fbb77b0a3f8d240c9924f3fa4529f5b5c8122 ("dist: debian: fix detection
of debuild") did in scylla-tools.
Message-Id: <20200911164153.34699-1-penberg@scylladb.com>
We do not want to install dependencies on package building time,
we want to install it on dbuild container.
So drop package installation from the scripts.
scylladb/scylla#7219
"
Fixes#133Fixes#134
Refs #135
Makes CF mbean refresh code synchronized and tries to remove reductant
calls if we contend. Adds background reaping of dead objects to reduce
memory load in (test) scenarios where we manage to refresh to add, but
not cause removal (i.e. no wildcard queries).
TableMetricsObjectName serialization is fixed in the series because
without it we see loads of exceptions when refreshing the mbean set.
"
* elcallio-jmx-fixes:
scylla-jmx: Use registration checker objects
scylla-jmx: Introduce a registration check object
scylla-jmx: Fix TableMetricObjectName serialization
Fixes#134
Refs #135
Replaces previous refresh calls with ones bound to registration
check objects, which provides some sync between threads doing
refresh, and reduced redundant calls.
Also adds repeated reaping of dead objects, i.e. every 5 minutes
we try to remove dead CF:s (not adding new ones), to reduce
idle footprint.
"Currently after offline installation, the scylla-jmx fails to start.
This pull request fixes issues with openjdk version detection and
working directory configuration to make scylla-jmx start.
Fixes: scylladb/scylla#7098 by [PATCH] install.sh: check both openjdk-8 and openjdk-11
Fixes: #129 by [PATCH] nonroot.conf: set WorkingDirectory to empty"
Reviewed-by: Takuya ASADA <syuu@scylladb.com>
* 'openjdk' of git://github.com/amoskong/scylla-jmx:
install.sh: check both openjdk-8 and openjdk-11
nonroot.conf: set WorkingDirectory to empty
Fixes#133
TableMetricObjectName is not serializable as such, since
it depends on a lexicon object etc.
Use writeReplace to put a regular ObjectName in
the stream instead.
On Debian10, only Openjdk-11 is available, the install.sh fails in java
checking. Openjdk-8 and Openjdk-11 all work well, we should check both of them.
This patch also fixed the error message.
Signed-off-by: Amos Kong <amos@scylladb.com>
After offline installation, scylla-jmx fails to be started for a chdir
error. WorkingDirectory is set to /var/lib/scylla in scylla-jmx.service,
it doesn't exist in nonroot install. This patch solved the problem by
setting WorkingDirectory to empty in nonroot.conf.
$ systemctl --user status scylla-jmx
● scylla-jmx.service - Scylla JMX
Loaded: loaded (/home/scylla-test/.config/systemd/user/../../../scylladb/etc/systemd/scylla-jmx.service; linked; vendor preset: enabled)
Drop-In: /home/scylla-test/.config/systemd/user/scylla-jmx.service.d
└─nonroot.conf
Active: failed (Result: exit-code) since Wed 2020-08-26 15:19:56 UTC; 2s ago
Process: 66955 ExecStart=/home/scylla-test/install_root/jmx/scylla-jmx $SCYLLA_JMX_PORT $SCYLLA_API_PORT $SCYLLA_API_ADDR $SCYLLA_JMX_ADDR $SCYLLA_JMX_FILE $SCYLLA_JMX_LOCAL $SCYLLA_JMX_REMOTE $SCYLLA_JMX_DEBUG (code=exited, status=200/CHDIR)
Main PID: 66955 (code=exited, status=200/CHDIR)
systemd[5654]: Started Scylla JMX.
systemd[66955]: scylla-jmx.service: Changing to the requested working directory failed: No such file or directory
systemd[66955]: scylla-jmx.service: Failed at step CHDIR spawning /home/scylla-test/scylladb/jmx/scylla-jmx: No such file or directory
systemd[5654]: scylla-jmx.service: Main process exited, code=exited, status=200/CHDIR
systemd[5654]: scylla-jmx.service: Failed with result 'exit-code'.
Signed-off-by: Amos Kong <amos@scylladb.com>
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
We current require Java 8 to install the scylla-jmx package on Debian.
As Debian 10 defaults to Java 11, let's remove the conflict flag and add
Java 11 to the dependencies list.
Currently scylla-jmx.service's PWD is "/", we get following error when
JVM trying to write heap dump on current directory:
Aug 17 05:52:15 localhost.localdomain scylla-jmx[3469]: Starting the JMX server
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: java.lang.OutOfMemoryError: Java heap space
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Dumping heap to java_pid3469.hprof ...
Aug 17 05:52:16 localhost.localdomain scylla-jmx[3469]: Unable to create java_pid3469.hprof: Permission denied
To fix this, we need to specify WorkingDirectory on systemd unit.
Debian package builds provide a root environment for the installation
scripts, since that's what typical installation scripts expect. To
avoid providing actual root, a "fakeroot" system is used where syscalls
are intercepted and any effect that requires root (like chown) is emulated.
However, fakeroot sporadically fails for us, aborting the package build.
Since our install scripts don't really require root (when operating in
the --packaging mode), we can just tell dpkg-buildpackage that we don't
need fakeroot. This ought to fix the sporadic failures.
As a side effect, package builds are faster.
Follows scylla.git's b608af870b0a1ad88b91a72bddeff0c321877f9e.
Refs scylladb/scylla#6655.
This directory is created by a build and shouldn't be commited so
it's best for git to just ignore it.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <b20dba12fb726aebd51b2ab9494e7c52f8058feb.1595259605.git.piotr@scylladb.com>
We need the ability to control build directory in scylla.git build
system. Let's add support for the "--builddir" option like in other
variants of the same scripts.
Message-Id: <20200717085723.701209-1-penberg@scylladb.com>
rpmbuild defaults to logging the name of every file it unpacks from
the archive. This is quite a lot for Java applications.
Make it quiet with the %setup -q flag.
When building scylla-jmx in place, `scylla-apiclient/target/` is left
behind and should be ignored by `.gitignore` otherwise the scylla
submodule directory appears to be dirty.
* bhalevy-gitignore:
gitignore: do not track scylla-apiclient/target/
gitignore: fix typo in dependency-reduced-pom.xml
