Packages

c

cloudflow.spark

SparkStreamletContext

abstract case class SparkStreamletContext(streamletDefinition: StreamletDefinition, session: SparkSession) extends StreamletContext with Product with Serializable

Linear Supertypes
Serializable, Serializable, Product, Equals, StreamletContext, AnyRef, Any
Known Subclasses
SparkStreamletContextImpl, TestSparkStreamletContext
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. SparkStreamletContext
  2. Serializable
  3. Serializable
  4. Product
  5. Equals
  6. StreamletContext
  7. AnyRef
  8. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SparkStreamletContext(streamletDefinition: StreamletDefinition, session: SparkSession)

Type Members

  1. case class MountedPathUnavailableException extends Exception with Product with Serializable
    Definition Classes
    StreamletContext

Abstract Value Members

  1. abstract def checkpointDir(dirName: String): String

    Returns the absolute path to a mounted shared storage that can be used to store reliable checkpoints.

    Returns the absolute path to a mounted shared storage that can be used to store reliable checkpoints. Reliable checkpoints lets the Spark application persist its state across restarts and restart from where it last stopped.

    returns

    the absolute path to a mounted shared storage

  2. abstract def readStream[In](inPort: CodecInlet[In])(implicit encoder: Encoder[In], typeTag: scala.reflect.api.JavaUniverse.TypeTag[In]): Dataset[In]

    Stream from the underlying external storage and return a DataFrame

    Stream from the underlying external storage and return a DataFrame

    inPort

    the inlet port to read from

    returns

    the data read as Dataset[In]

  3. abstract def writeStream[Out](stream: Dataset[Out], outPort: CodecOutlet[Out], outputMode: OutputMode, trigger: Option[Trigger])(implicit encoder: Encoder[Out], typeTag: scala.reflect.api.JavaUniverse.TypeTag[Out]): StreamingQuery

    Start the execution of a StreamingQuery that writes the encodedStream to an external storage using the designated portOut

    Start the execution of a StreamingQuery that writes the encodedStream to an external storage using the designated portOut

    stream

    stream used to write the result of execution of the StreamingQuery

    outPort

    the port used to write the result of execution of the StreamingQuery

    outputMode

    the output mode used to write. Valid values Append, Update, Complete

    trigger

    Execution trigger (see http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#triggers for details)

    returns

    the StreamingQuery that starts executing

Concrete Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  4. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  5. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native() @HotSpotIntrinsicCandidate()
  6. def config: Config
    Definition Classes
    StreamletContext
  7. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  8. def findTopicForPort(port: StreamletPort): Topic
    Definition Classes
    StreamletContext
  9. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  10. def getMountedPath(volumeMount: VolumeMount): Path
    Definition Classes
    StreamletContext
  11. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  12. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  13. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  14. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native() @HotSpotIntrinsicCandidate()
  15. def runtimeBootstrapServers(topic: Topic): String
    Definition Classes
    StreamletContext
  16. val session: SparkSession
  17. final def streamletConfig: Config
    Definition Classes
    StreamletContext
  18. def streamletRef: String
    Definition Classes
    StreamletContext
  19. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  20. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  21. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  22. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Deprecated Value Members

  1. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] ) @Deprecated
    Deprecated

Inherited from Serializable

Inherited from Serializable

Inherited from Product

Inherited from Equals

Inherited from StreamletContext

Inherited from AnyRef

Inherited from Any

Ungrouped