Currently, we have a scary looking dtest failure when attempting to force flush a
Nodetool command '/data/jenkins/workspace/scylla-1.3-dtest/label/monster/mode/release/smp/1/scylla/resources/cassandra/bin/nodetool -h localhost -p 7100 flush' failed; exit status: 2; stderr: Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
error: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
-- StackTrace --
java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:245)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
at com.sun.proxy.$Proxy7.forceKeyspaceFlush(Unknown Source)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceFlush(NodeProbe.java:290)
at org.apache.cassandra.tools.NodeTool$Flush.execute(NodeTool.java:1227)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
The problem is rather innocent: the API call fails and we leak
javax.ws.rs.ProcessingException, which is not available in nodetool's
classpath. In fact, we already fixed the problem for GETs in commit
02e0598 ("APIClient: Fix error handling if connection to API server
fails") so do the same thing for POSTs.
Message-Id: <1471589525-26435-1-git-send-email-penberg@scylladb.com>
Ran into this while trying to use ccm with not up-to-date scylla-jmx.
Symptoms:
$ ccm start
Error starting node node1
and empty ~/.ccm/scylla-3/node1/logs/system.log.jmx
Message-Id: <1468399926-3565-1-git-send-email-tgrabiec@scylladb.com>
The correct dependency between the jmx and the scylla-server is:
The scylla-jmx should not run if the scylla-server is not running, it
should shutdown when the scylla-server shuts down.
Starting the scylla-jmx should not start the scylla-server, instead, if
the scylla-server is not running it should fail to start.
This patch changes the setup to do so.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1467184319-3395-1-git-send-email-amnon@scylladb.com>
The batch mode produces much more readable logs because it's designed
for non-interactive builds and doesn't have the fancy download progress
meters.
Message-Id: <1464770158-32482-1-git-send-email-penberg@scylladb.com>
Commit 12daaf5 ("dist/redhat: fix rpm build error") did not fix the
error, at least not on our Jenkins build machines.
Looking at the RPM build logs, we create the build directory:
+ cd /builddir/build/BUILD
+ mkdir build
but then change directory to "scylla-jmx-1.2.rc1":
+ cd /builddir/build/BUILD
+ cd scylla-jmx-1.2.rc1
+ mvn install
and therefore fail the copy:
+ cp dist/common/systemd/scylla-jmx.service.in build/scylla-jmx.service
cp: cannot create regular file 'build/scylla-jmx.service': No such
file or directory
I don't know why Takuya put the "mkdir" in the "prep" section but
something like this should unblock the build.
When removing the pull based timers in the API the sum method in the
APITimer was left stubed by mistake.
This patch take the sum from the histogram as it should be.
Another missed changes are the units, in the yammer library the Timer
does unit conversion before returning the values.
This patch takes the unit conversion from the yammer library to be
compatible.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1464173726-7482-1-git-send-email-amnon@scylladb.com>
Since build/ is not exist, 'cp dist/common/systemd/scylla-jmx.service.in build/scylla-jmx.service' will fail.
So create build/ before starting build stage.
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1464164524-20867-1-git-send-email-syuu@scylladb.com>
"This series uses the API that was added to scylla to remove the counter
pulling and rely on the statistics collected by the API.
The series extends the APIMeter, APIHistogram and APITimer to remove
their pulling part and to fetch the information when needed from the
API.
For performence reason those objects will be cached, so that in the
typical case of of multiple requests of different fields will cause a
single API call."
Since Ubuntu 15.10/16.04 has moved to systemd, share CentOS/Fedora's systemd unit file with Ubuntu.
Fixesscylladb/scylla#1283
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1464083473-1701-1-git-send-email-syuu@scylladb.com>
Helps in top, pgrep and friends. Unfortunately the only reasonable way
to do it is to create a symlink to /usr/bin/java and run that.
Message-Id: <1463580254-8369-1-git-send-email-avi@scylladb.com>
With the change to APITimer there is no longer a need to periodically
pull the API.
The verb will be register on the object initialization and will be
updated whenever they are been used.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
With the change in the meter implementation, retrieving a non existing
metrics would take time.
For this, the CacheService would mark caches that are not supported with
null url, so the metrics will be register but will return 0 for all
request (instead of going to the API that will return 0).
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
CacheMetrics is a general counter that is used for all possible caches.
For caches that we do not support, there is no need to go and fetch
their values.
When moving to the APITimer each such request will take longer (the
value will not be available as it use it) now it will be possible to
supply a null as a url which would cause to the metrics to return 0 for
all counters.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The APITimer uses a different endpoint not to break existing API.
The addNano functionality was removed as all of the values are updated
from the APi.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Some of the specific functionality is needed from the APIMeter and
APITimer.
The MetricsRegistry now return the specific objects so they can be used
in their adapted form.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The yammer Timer object calculate rate based on a timer, which causes
periodic calls to the API.
This replaces the implementation so that a timer would get all its
values from the API.
For object registration the APITimer still inherit from Timer but
override all its functionality.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
With the move to APITimer, in many occasion a histogram will not update
itself, instead it will be updated by the APITimer.
This breaks the update values functionality so that a histogram that is
included in an APITimer will not try to update it self.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This replaces the APIMeter implementation so it will not pull the API
regularly.
To by complient with the yammer object registration mechanism, it still
inherit from Meter, but all the functionalities are overriden.
Now all the data is taken from the API including the rate statistics.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
APIMeter will be modified not to use pulling and to retrieve the derived
information from the API.
To support that the APIClient was changed so it would be able to cache
json objects and histogram.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
When setting the jmx through the command line, the jmx server creates
even before the main is called.
For the APIServer to take effect the builder should be set via system
properties.
This patch also add an option to run the java process with debug ports
open so an extern debug will be able to connect to the app.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1462447227-8367-2-git-send-email-amnon@scylladb.com>
The mx4j implementation of queryName does not handle correctly pattern
matching.
This patch identify that a name contains a patern and do the patern
matching as it should have been done by the mx4j MBeanServer.
Fixes#28
Message-Id: <1462447227-8367-1-git-send-email-amnon@scylladb.com>
"This series replaces that mechanism with an implementation of the MBeanServer
that intercept the relevant MBean call and call the relevant registration
function.
The pulling mechanism was removed from Main."
With the addition of the APIMBeanServer there is no longer a need for
the pulling functionality to be perform for MBean registration.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This register the APIBuilder as the MBeanServerBuilder which will cause
the APIMBeanServer to be used as the MBeanServer.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The APIBuilder is an implementation for the MBeanServerBuilder that is
used to instantiate the APIMBeanServer as the platform MBeanServer.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The APIMBeanServer is serve as a proxy for the MBeanServer.
It intercept calls to the MBeanServer and check for the column family
and stream registeration before they are perform.
Current implementation override queryNames as it's the one that is being
used by nodetool.
Additional methods can be override in the future if needed.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The MX4J introspector does not support *MXBean interfaces name, This
causes a problem with the garbage collector and java related MBeans.
To bypass that limitation the APIMBeanIntrospector inherit from
MBeanIntrospector and override the relevant functionality so MXBean
will be treated like MBean.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This expose the ColumnFamilyStore registration via static method.
It would allow an external object (ie. MBeanServer) to update the
registration on demand.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This patch adds the functionality of takeMultipleColumnFamilySnapshot to
StorageService.
It follow origin logic of first check that all keyspaces and column
families exists and has no snapshot with that name and then run snapshot
on each of the combinations.
Two methods where added to simplify the implementation, but that can be
reused. One to get a map from keyspace to column family and one with the
current snapshots.
Fixesscylladb/scylla#1133
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1461659678-22030-1-git-send-email-amnon@scylladb.com>
During upgrade or version inconsistency. The API can return an un
supported state.
Instead of throwing an expcetion the state will be ignore and a warning
will be written to the log.
An example (state where modified in the API)
$ nodetool gossipinfo
/127.0.0.1
generation:1460450456
heartbeat:32
The log shows:
Apr 12, 2016 3:40:20 PM org.apache.cassandra.gms.EndpointState
addApplicationState
WARNING: Unknown application state with id:25
Fixesscylladb/scylla#1164.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1460465073-3567-1-git-send-email-amnon@scylladb.com>
Drop sysv init script on scylla-jmx.
Same as a5bb6c4b1b
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459346746-3433-2-git-send-email-syuu@scylladb.com>
Same as f1d18e9980Fixesscylladb/scylla#1134
Signed-off-by: Takuya ASADA <syuu@scylladb.com>
Message-Id: <1459346746-3433-1-git-send-email-syuu@scylladb.com>
The version number ordering rules are different for rpm and deb. Use
tilde ('~') for the latter to ensure a release candidate is ordered
_before_ a final version.
When getting the tokens of the current node, we use the get_token api
call with the local broadcast address.
The current implementation that tries to figure it out from the
configuration is prone to error.
Currently in a configuration where the broadcast address is set to the
local API and the listening API is set to 127.0.0.1 we get a call to
nodetool info will return an exception:
ID : 54185d5d-6f62-4884-814c-5d17c2776de9
Gossip active : true
Thrift active : true
Native Transport active: true
Load : 178.09 KB
Generation No : 1458349593
Uptime (seconds) : 11
Heap Memory (MB) : 47.23 / 247.50
Off Heap Memory (MB) : 2.75
error: Index: 0, Size: 0
-- StackTrace --
java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.rangeCheck(ArrayList.java:653)
at java.util.ArrayList.get(ArrayList.java:429)
at org.apache.cassandra.tools.NodeProbe.getEndpoint(NodeProbe.java:812)
at org.apache.cassandra.tools.NodeProbe.getDataCenter(NodeProbe.java:830)
at org.apache.cassandra.tools.NodeTool$Info.execute(NodeTool.java:425)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
Becasue getTokens will return an empty list.
This patch changed how broadcast address is deduct. It Adds a reverse
mapping from hostid to ip address and use it with the get local id to
find the ip address in use.
This implementation would probably be replaced by a single API call in
the future.
After the change a call to nodetool info works.
Fixesscylladb/scylla#1027
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1458405434-8491-3-git-send-email-amnon@scylladb.com>
