大多数场景下，大数据环境会使用Kerberos进行安全加固。正如配置文件livy.conf.template所描述的Livy支持的认证方式有Kerberos和自定义认证。
# Authentication support for Livy server
# Livy has a built-in SPnego authentication support for HTTP requests with below configurations.
# livy.server.auth.type = kerberos
# livy.server.auth.kerberos.principal = <spnego principal>
# livy.server.auth.kerberos.keytab = <spnego keytab>
# livy.server.auth.kerberos.name-rules = DEFAULT
#
# If user wants to use custom authentication filter, configurations are:
# livy.server.auth.type = <custom>
# livy.server.auth.<custom>.class = <class of custom auth filter>
# livy.server.auth.<custom>.param.<foo1> = <bar1>
# livy.server.auth.<custom>.param.<foo2> = <bar2>
本博客将简单介绍如何给Livy实现一个使用用户名和密码的认证方式。
1 源码实现逻辑分析
1.1 认证入口
org.apache.livy.server.LivyServer
livyConf.get(AUTH_TYPE) match {
**** 中间省略
case customType =>
val authClassConf = s"livy.server.auth.$customType.class"
val authClass = livyConf.get(authClassConf)
require(authClass != null, s"$customType auth requires $authClassConf to be provided")
val holder = new FilterHolder()
holder.setClassName(authClass)
val prefix = s"livy.server.auth.$customType.param."
livyConf.asScala.filter { kv =>
kv.getKey.length > prefix.length && kv.getKey.startsWith(prefix)
}.foreach { kv => holder.setInitParameter(kv.getKey.substring(prefix.length), kv.getValue) }
server.context.addFilter(holder, "/*", EnumSet.allOf(classOf[DispatcherType]))
info(s"$customType auth enabled")
需要关注里面用到的3个配置
- customType 对应配置livy.server.auth.type
- livy.server.auth.$customType.class 我们使用的Filter
- livy.server.auth.$customType.param. 可定义多个参数
1.2 class选择和param传递
和kerberos认证一样，选择org.apache.hadoop.security.authentication.server
val holder = new FilterHolder(new AuthenticationFilter())
里面比较核心的一段代码，AuthenticationFilter对象调用authHandler中的authenticate方法。
public void doFilter(ServletRequest request, ServletResponse response, FilterChain filterChain)
...
if (authHandler.managementOperation(token, httpRequest, httpResponse)) {
if (token == null) {
if (LOG.isDebugEnabled()) {
LOG.debug("Request [{}] triggering authentication", getRequestURL(httpRequest));
}
token = authHandler.authenticate(httpRequest, httpResponse);
if (token != null && token.getExpires() != 0 &&
token != AuthenticationToken.ANONYMOUS) {
token.setExpires(System.currentTimeMillis() + getValidity() * 1000);
}
newToken = true;
}
authHandler在此处创建，其中AUTH_TYPE=type
public void init(FilterConfig filterConfig) throws ServletException {
String configPrefix = filterConfig.getInitParameter(CONFIG_PREFIX);
configPrefix = (configPrefix != null) ? configPrefix + "." : "";
config = getConfiguration(configPrefix, filterConfig);
String authHandlerName = config.getProperty(AUTH_TYPE, null);
String authHandlerClassName;
if (authHandlerName == null) {
throw new ServletException("Authentication type must be specified: " +
PseudoAuthenticationHandler.TYPE + "|" +
KerberosAuthenticationHandler.TYPE + "|<class>");
}
if (authHandlerName.toLowerCase(Locale.ENGLISH).equals(
PseudoAuthenticationHandler.TYPE)) {
authHandlerClassName = PseudoAuthenticationHandler.class.getName();
} else if (authHandlerName.toLowerCase(Locale.ENGLISH).equals(
KerberosAuthenticationHandler.TYPE)) {
authHandlerClassName = KerberosAuthenticationHandler.class.getName();
} else {
authHandlerClassName = authHandlerName;
}
参数传递逻辑如下，其中也包含上面会用到的type
val prefix = s"livy.server.auth.$customType.param."
livyConf.asScala.filter { kv =>
kv.getKey.length > prefix.length && kv.getKey.startsWith(prefix)
}.foreach { kv =>
holder.setInitParameter(kv.getKey.substring(prefix.length), kv.getValue)
}
server.context.addFilter(holder, "/*", EnumSet.allOf(classOf[DispatcherType]))
2 认证开发和配置
通过上面的逻辑分析，我们知道需要开发的内容是type对应的class（需要继承AuthenticationHandler），并且可以在param定义class所需的各种参数。
/**
* @author xxx
* @date 2023/3/9 5:33 PM
*/
class LivyUserAuth extends AuthenticationHandler {
val log = LoggerFactory.getLogger(this.getClass)
private var userPassList:mutable.Map[String, String] = mutable.HashMap()
private var hdfsLocation:String = null
private var fileLocation:String = null
private var periodSecond:Int = -1
private var decryptType:String = null
private val CACHE_SIZE:Int = 1000
private var passwordSupplier:Supplier[PasswordStore] = null
override def getType: String = "xxx"
override def init(properties: Properties): Unit = {
}
override def destroy(): Unit = {
}
/**
* return true when the request processing should continue. If false is returned, it means the response has been populated by this method
* @param authenticationToken
* @param httpServletRequest
* @param httpServletResponse
*/
@throws[IOException]
@throws[AuthenticationException]
override def managementOperation(authenticationToken: AuthenticationToken,
httpServletRequest: HttpServletRequest,
httpServletResponse: HttpServletResponse): Boolean = {
return true
}
@throws[IOException]
@throws[AuthenticationException]
override def authenticate(request: HttpServletRequest,
response: HttpServletResponse): AuthenticationToken = {
var token: AuthenticationToken = null
var authorization = request.getHeader("Authorization")
log.info("authorization:" + authorization)
token
}
}
livy.conf新增以下配置
livy.server.auth.type nxb
livy.server.auth.nxb.class org.apache.hadoop.security.authentication.server.AuthenticationFilter
livy.server.auth.nxb.param.file.location xxxxxxxxxx
livy.server.auth.nxb.param.period.second 1800
livy.server.auth.nxb.param.type com.nxb.livy.auth.LivyUserAuth
3 打包和部署
只需要将项目打包即可，依赖不需要（Livy Jars里面已经有了）
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<outputDirectory>${project.build.directory}/jars</outputDirectory>
</configuration>
</plugin>
</plugins>
将打包好jar放在/usr/lib/livy/jars/
重启服务
- sudo systemctl restart livy-server.service
