我想通過僅使用向量中的前 k 個值來實現使 softmax 更快的嘗試。為此，我嘗試為 tensorflow 實現一個自定義函數以在模型中使用：def softmax_top_k(logits, k=10): values, indices = tf.nn.top_k(logits, k, sorted=False) softmax = tf.nn.softmax(values) logits_shape = tf.shape(logits) return_value = tf.sparse_to_dense(indices, logits_shape, softmax) return_value = tf.convert_to_tensor(return_value, dtype=logits.dtype, name=logits.name) return return_value我正在使用時尚 mnist 來測試該嘗試是否有效：fashion_mnist = keras.datasets.fashion_mnist(train_images, train_labels), (test_images, test_labels) = fashion_mnist.load_data()# normalize the datatrain_images = train_images / 255.0test_images = test_images / 255.0# split the training data into train and validate arrays (will be used later)train_images, train_images_validate, train_labels, train_labels_validate = train_test_split( train_images, train_labels, test_size=0.2, random_state=133742,)model = keras.models.Sequential([ keras.layers.Flatten(input_shape=(28, 28)), keras.layers.Dense(128, activation=tf.nn.relu), keras.layers.Dense(10, activation=softmax_top_k)])model.compile( loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])model.fit( train_images, train_labels, epochs=10, validation_data=(train_images_validate, train_labels_validate),)model_without_cnn.compile( loss='sparse_categorical_crossentropy', optimizer='adam', metrics=['accuracy'])model_without_cnn.fit( train_images, train_labels, epochs=10, validation_data=(train_images_validate, train_labels_validate),)但是在執行過程中出現錯誤：ValueError: An operation has沒有任何for gradient. Please make sure that all of your ops have a gradient defined (i.e. are differentiable).我發現了這個 : (How to make a custom activation function)，它解釋了如何對 tensorflow 實現完全自定義的激活函數。但是由于這使用并擴展了 softmax，我認為梯度應該仍然相同。這是我使用 python 和 tensorflow 進行編碼的第一周，因此我還沒有對所有內部實現有一個很好的概述。有沒有更簡單的方法將 softmax 擴展到新函數中，而不是從頭開始實現？提前致謝！
1 回答
慕妹3242003
TA貢獻1824條經驗 獲得超6個贊
不是使用稀疏張量來制作“除 softmaxed top-K 值之外的所有零”的張量，而是使用tf.scatter_nd：
import tensorflow as tf
def softmax_top_k(logits, k=10):
values, indices = tf.nn.top_k(logits, k, sorted=False)
softmax = tf.nn.softmax(values)
logits_shape = tf.shape(logits)
# Assuming that logits is 2D
rows = tf.tile(tf.expand_dims(tf.range(logits_shape[0]), 1), [1, k])
scatter_idx = tf.stack([rows, indices], axis=-1)
return tf.scatter_nd(scatter_idx, softmax, logits_shape)
編輯：這是具有任意維數的張量的稍微復雜的版本。但是，代碼仍然要求在圖構建時知道維數。
import tensorflow as tf
def softmax_top_k(logits, k=10):
values, indices = tf.nn.top_k(logits, k, sorted=False)
softmax = tf.nn.softmax(values)
# Make nd indices
logits_shape = tf.shape(logits)
dims = [tf.range(logits_shape[i]) for i in range(logits_shape.shape.num_elements() - 1)]
grid = tf.meshgrid(*dims, tf.range(k), indexing='ij')
scatter_idx = tf.stack(grid[:-1] + [indices], axis=-1)
return tf.scatter_nd(scatter_idx, softmax, logits_shape)
添加回答
舉報
0/150
提交
取消