This is a seemingly pointless change. The RW-lock code is 100%
correct (afaict), yet we've seen repeated cases of test runs
hanging in JMX query because this lock is seemingly left held
by what seems to be the reaper task.
There is no explanation for this, no sign of exceptions/errors
that could explain the lock being broken. Nor any known JDK/JVM
bugs.
Yet, in tests, it seems that replacing the lock with a more
coarse, yet proven, synchronized, fixes the issue. So there.
I officially hate this patch, and it should not exist.
Fixes#134
Refs #135
Replaces previous refresh calls with ones bound to registration
check objects, which provides some sync between threads doing
refresh, and reduced redundant calls.
Also adds repeated reaping of dead objects, i.e. every 5 minutes
we try to remove dead CF:s (not adding new ones), to reduce
idle footprint.
delete commands do not return a value, still, it is possible that the
command will return a value different than OK.
In such a case, the error should be propagate to the caller via an
exception.
Fixes#65
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <20190618135312.2776-1-amnon@scylladb.com>
Fixes#57
The usage of TableMetricsObjectName-yada-yada relies on translating the
"fake" objectname to a canonical one on remote
publication/serialization. However, the implementation of
ObjectName.getInstance has changed in JDK (micro) updates so it no
longer applies overridable methods -> wrong name published.
Fix by doing explicit ObjectName instansiation.
Message-Id: <20181023132005.23099-1-calle@scylladb.com>
Before we were discarding the initial repository while
overriding it with TableRepository. This was a mistake that
caused dtests to fail. Proper solution is to keep the initial
repository inside TableRepository. That way whatever was registered
at the time of JmxMBeanServer creation is still handled properly.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Message-Id: <22181859012fd20ddf37e049a145bc94a3a91a33.1527844328.git.piotr@scylladb.com>
Default implementation stores MBeans in the following map:
<domain name> -> (<properties as a single string> -> NamedObject)
This is problematic because NamedObject contains ObjectName that
has both domain and properties inside itself.
This means we're storing the same data twice.
For domain "" we want to store MBeans in a more compact way using map:
ObjectName -> DynamicMBean
which is equivalent to NamedObject.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Before this change it was taking JMX Server 270 seconds to start
when Scylla had 2000 tables. After the change it takes only 2 seconds.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
Next patch will introduce new ObjectName implementation that
will use less memory. This new object won't be serializable.
This means it won't be possible to transport it to a remote
caller. We want to keep this new object local to JMX server as well.
This patch makes sure that every ObjectName returned
from APIBeanServer is transformed into a regular ObjectName.
It also makes sure that every ObjectInstance returned from
APIBeanServer has its ObjectName swapped with a regular ObjectName.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
JmxMBeanServer is a concrete implementation of a MBeanServer.
We want to use it directly because we need to bypass calls to
JmxMBeanServer.registerMBean and JmxMBeanServer.unregisterMBean.
They take ObjectName as parameter, copy it using
ObjectName.getInstance(ObjectName) and pass it to registerMBean
and unregisterMBean of JmxMBeanServer.getMBeanServerInterceptor().
We want to avoid this copy and pass the ObjectName argument directly
to JmxMBeanServer.getMBeanServerInterceptor().
To do that this patch:
1. changes all MBeanServer variables to JmxMBeanServer
2. creates JmxMBeanServer in APIBuilder making sure accessing
interceptors is allowed
3. makes sure that JmxMBeanServer.getMBeanServerInterceptor().registerMBean
is called wherever JmxMBeanServer.registerMBean was called
4. makes sure that JmxMBeanServer.getMBeanServerInterceptor().unregisterMBean
is called whenever JmxMBeanServer.unregisterMBean was called
Next patch will use different ObjectName implementation that will
use less memory and this patch is crucial because without it every ObjectName
is transformed with ObjectName.getInstance which turns the object into
a regular ObjectName.
Signed-off-by: Piotr Jastrzebski <piotr@scylladb.com>
When parsing the snapshot disk sizes, it should be long and not int.
See scylladb/scylla#2104
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1487760019-1354-1-git-send-email-amnon@scylladb.com>
Actually removes a bunch of code to manage the JMX connector,
since as of jdk8u102, the standard jmx connector answers to
property setting bind address -> can restrict access.
Note that the RMI connector will now (as is jdk normal)
_bind_ to 0.0.0.0, but it will not answer non-local requests
if "remote" is not enabled. This is the default jdk behaviour.
In any case, we rely on setting the appropriate properties
instead, and also allow pass-through of -D flags to java,
which in turn means those who wish can turn on both auth
and ssl, set key/trust stores etc etc.
Message-Id: <1485357178-20714-1-git-send-email-calle@scylladb.com>
The getMapStrValue return a map from the API. This change the
implementation to use linkedHashMap so the map will be sorted according
to the API order.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Currently, we have a scary looking dtest failure when attempting to force flush a
Nodetool command '/data/jenkins/workspace/scylla-1.3-dtest/label/monster/mode/release/smp/1/scylla/resources/cassandra/bin/nodetool -h localhost -p 7100 flush' failed; exit status: 2; stderr: Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
error: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
-- StackTrace --
java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:396)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:186)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:219)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1620)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1521)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:245)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:162)
at com.sun.jmx.remote.internal.PRef.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnectionImpl_Stub.invoke(Unknown Source)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.invoke(RMIConnector.java:1020)
at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:298)
at com.sun.proxy.$Proxy7.forceKeyspaceFlush(Unknown Source)
at org.apache.cassandra.tools.NodeProbe.forceKeyspaceFlush(NodeProbe.java:290)
at org.apache.cassandra.tools.NodeTool$Flush.execute(NodeTool.java:1227)
at org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:288)
at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
The problem is rather innocent: the API call fails and we leak
javax.ws.rs.ProcessingException, which is not available in nodetool's
classpath. In fact, we already fixed the problem for GETs in commit
02e0598 ("APIClient: Fix error handling if connection to API server
fails") so do the same thing for POSTs.
Message-Id: <1471589525-26435-1-git-send-email-penberg@scylladb.com>
Some of the specific functionality is needed from the APIMeter and
APITimer.
The MetricsRegistry now return the specific objects so they can be used
in their adapted form.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
APIMeter will be modified not to use pulling and to retrieve the derived
information from the API.
To support that the APIClient was changed so it would be able to cache
json objects and histogram.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The mx4j implementation of queryName does not handle correctly pattern
matching.
This patch identify that a name contains a patern and do the patern
matching as it should have been done by the mx4j MBeanServer.
Fixes#28
Message-Id: <1462447227-8367-1-git-send-email-amnon@scylladb.com>
With the addition of the APIMBeanServer there is no longer a need for
the pulling functionality to be perform for MBean registration.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This register the APIBuilder as the MBeanServerBuilder which will cause
the APIMBeanServer to be used as the MBeanServer.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The APIBuilder is an implementation for the MBeanServerBuilder that is
used to instantiate the APIMBeanServer as the platform MBeanServer.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The APIMBeanServer is serve as a proxy for the MBeanServer.
It intercept calls to the MBeanServer and check for the column family
and stream registeration before they are perform.
Current implementation override queryNames as it's the one that is being
used by nodetool.
Additional methods can be override in the future if needed.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The MX4J introspector does not support *MXBean interfaces name, This
causes a problem with the garbage collector and java related MBeans.
To bypass that limitation the APIMBeanIntrospector inherit from
MBeanIntrospector and override the relevant functionality so MXBean
will be treated like MBean.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Sometimes it is usefull to get a reverse of the map return by the API.
For example instread of ip address to hostid to get the hostid to ip
address.
Though it is possible to reverse a map, there is no need, it's easier to
generate the reverse mapping.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Message-Id: <1458405434-8491-2-git-send-email-amnon@scylladb.com>
Make the error messages returned by Scylla API server human readable
from 'nodetool'.
For example, if an API URL is missing, print out the following error:
[penberg@nero scylla-tools-java]$ ./bin/nodetool getcompactionthreshold ks test4
nodetool: Scylla API server HTTP GET to URL 'column_family/minimum_compaction/ks:test4' failed: Not found
See 'nodetool help' or 'nodetool help <command>'.
instead of the scary-looking error that we now print:
[penberg@nero scylla-tools-java]$ ./bin/nodetool getcompactionthreshold ks test4
error: Not found
-- StackTrace --
java.lang.RuntimeException: Not found
at com.scylladb.jmx.api.APIClient.getException(APIClient.java:116)
at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:160)
at com.scylladb.jmx.api.APIClient.getRawValue(APIClient.java:174)
at com.scylladb.jmx.api.APIClient.getIntValue(APIClient.java:216)
at com.scylladb.jmx.api.APIClient.getIntValue(APIClient.java:220)
at org.apache.cassandra.db.ColumnFamilyStore.getMinimumCompactionThreshold(ColumnFamilyStore.java:475)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[snip]
Message-Id: <1458032300-17704-1-git-send-email-penberg@scylladb.com>
Running 'nodetool status' now reports the following if the JMX proxy is
not able to connect to an API server:
nodetool: Unable to connect to Scylla API server: java.net.ConnectException: Connection refused
See 'nodetool help' or 'nodetool help <command>'.
instead of the scary-looking:
error: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
-- StackTrace --
java.lang.ClassNotFoundException: javax.ws.rs.ProcessingException (no security manager: RMI class loader disabled)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:393)
at sun.rmi.server.LoaderHandler.loadClass(LoaderHandler.java:185)
at java.rmi.server.RMIClassLoader$2.loadClass(RMIClassLoader.java:637)
at java.rmi.server.RMIClassLoader.loadClass(RMIClassLoader.java:264)
at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:214)
That happens because the MBean propagates a
'javax.ws.rs.ProcessingException' to nodetool which does not have it in
it's classpath and loading via RMI fails.
Fixes#25.
Message-Id: <1457697628-31792-1-git-send-email-penberg@scylladb.com>
This patch init the jmx proxy from the RMIServerSocketFactoryImp init
function. This way the jmx can be set to listen on local port only.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
The RMIServerSocketFactoryImp is the way origin handle local port
configuration.
When used, the jmx can be set to listen on local traffic only.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
This patch adds a method to the APIClient that return a map of String
and Double.
It support both simple and with query parameters.
Signed-off-by: Amnon Heiman <amnon@scylladb.com>
Andrea Cavalli
Pekka Enberg
Calle Wilund
Calle Wilund
Amnon Heiman
Lubos Kosco
Piotr Jastrzebski
elcallio
Amnon Heiman